Original: Deep Combinatorial Aggregation
Revised: Deep Combinatorial Aggregation

Original: Abstract
Revised: Abstract

Original: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.
Revised: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.

Original: This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks.
Revised: This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks.

Original: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).
Revised: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).

Original: DCA creates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.
Revised: DCA creates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.

Original: DCA components can be deﬁned at different levels of granularity.
Revised: DCA components can be deﬁned at different levels of granularity.

Original: And we discovered that coarse-grain
Revised: And we discovered that coarse-grain

Original: DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation.
Revised: DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation.

Original: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training.
Revised: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training.

Original: It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers.
Revised: It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers.

Original: Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA.
Revised: Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA.

Original: We experiment on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, and empirically conﬁrm the effectiveness of DCWA and DCA approaches.
Revised: We experiment on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, and empirically conﬁrm the effectiveness of DCWA and DCA approaches.

Original:  Introduction
Revised:  Introduction

Original: Deep learning has achieved groundbreaking progress and neural networks are now widely used in various domains [27].
Revised: Deep learning has achieved groundbreaking progress and neural networks are now widely used in various domains [24].

Original: However, they are known to produce poor uncertainty estimations [11, 41, 12], which can be problematic for challenges like safety-critical applications [20, 29] or active learning [38].
Revised: However, they are known to produce poor uncertainty estimations [9, 37, 10], which can be problematic for challenges like safety-critical applications [17, 26] or active learning [35].

Original: Numerous approaches have been proposed to tackle this issue, among which an effective yet simple method is deep ensemble [26].
Revised: Numerous approaches have been proposed to tackle this issue, among which an effective yet simple method is deep ensemble [23].

Original: Deep ensemble yields state-of-the-art results for uncertainty aware learning [41, 12] and it does not require elaborate architectural design and hyperparameter search.
Revised: Deep ensemble yields state-of-the-art results for uncertainty aware learning [37, 10] and it does not require elaborate architectural design and hyperparameter search.

Original: However, while it aggregates multiple separately trained models, it can neither generate new samples from the posterior to obtain more diverse predictions, nor produce a summarizing average model which improves on the individual models in the ensemble.
Revised: However, while it aggregates multiple separately trained models, it can neither generate new samples from the posterior to obtain more diverse predictions, nor produce a summarizing average model which improves on the individual models in the ensemble.

Original: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation (DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neural networks, we explore the idea of ensembling components of the network architecture and combining them to form an enriched collection of model proposals.
Revised: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation (DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neural networks, we explore the idea of ensembling components of the network architecture and combining them to form an enriched collection of model proposals.

Original: DCA inherits the simplicity and effectiveness of deep ensemble, and additionally leads to several new possibilities: Apart from generating a diversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new average proposal via deep combinatorial weight averaging (DCWA).
Revised: DCA inherits the simplicity and effectiveness of deep ensemble, and additionally leads to several new possibilities: Apart from generating a diversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new average proposal via deep combinatorial weight averaging (DCWA).

Original: Furthermore, DCA training can beneﬁt from a consistency enforcing loss, which can produce DCA models that surpass standard deep ensemble both in classiﬁcation performance and uncertainty estimation.
Revised: Furthermore, DCA training can beneﬁt from a consistency enforcing loss, which can produce DCA models that surpass standard deep ensemble both in classiﬁcation performance and uncertainty estimation.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: Improving not only the predictive performance but also the uncertainty estimation of neural networks has been a core objective of Bayesian deep learning, where an abundance of prior work exists.
Revised: Improving not only the predictive performance but also the uncertainty estimation of neural networks has been a core objective of Bayesian deep learning, where an abundance of prior work exists.

Original: This includes methods based on variational inference and weight perturbation such as Bayes by backprop [10, 4] and its variants [45, 6], Bayesian interpretation of dropout including MC dropout [8] and variational dropout [3, 33], expectation propagation which leads to probabilistic backpropagation [16], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics (SGLD) [44] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [5], stochastic gradient descent (SGD) as approximate MCMC which results in SWAG [31], as well as Bayesian noisy optimizers such as variational online Gauss-Newton (VOGN) [21] and approaches using Laplace approximation [23, 36].
Revised: This includes methods based on variational inference and weight perturbation such as Bayes by backprop [8, 2] and its variants [41, 4], Bayesian interpretation of dropout including MC dropout [6] and variational dropout [1, 30], expectation propagation which leads to probabilistic backpropagation [14], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics (SGLD) [40] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [3], stochastic gradient descent (SGD) as approximate MCMC which results in SWAG [28], as well as Bayesian noisy optimizers such as variational online Gauss-Newton (VOGN) [18] and approaches using Laplace approximation [20, 33].

Original: Beyond the Bayesian formulation, methods like post-hoc calibration [11, 43] readjust trained networks to produce more calibrated predictions, while approaches like evidential deep learning [37] also make use of ideas like subjective logic.
Revised: Beyond the Bayesian formulation, methods like post-hoc calibration [9, 39] readjust trained networks to produce more calibrated predictions, while approaches like evidential deep learning [34] also make use of ideas like subjective logic.

Original: Most relevant to our work is the deep ensemble method [26], which aggregates multiple independently trained network models with different initial parameters.
Revised: Most relevant to our work is the deep ensemble method [23], which aggregates multiple independently trained network models with different initial parameters.

Original: Deep ensemble has been shown to produce state-of-the-art results for uncertainty estimation [41, 12].
Revised: Deep ensemble has been shown to produce state-of-the-art results for uncertainty estimation [37, 10].

Original: It can be combined with methods like MC dropout [7] and SWAG [48], or extended to the hyperparemeter space [47].
Revised: It can be combined with methods like MC dropout [5] and SWAG [44], or extended to the hyperparemeter space [43].

Original: Several variants have also been proposed, which often aim at providing more computationally or memory-efﬁcient alternatives.
Revised: Several variants have also been proposed, which often aim at providing more computationally or memory-efﬁcient alternatives.

Original: This includes snapshot ensemble [17], BatchEnsemble [46], fast geometric ensembling [9], TreeNet [28], etc .
Revised: This includes snapshot ensemble [15], BatchEnsemble [42], fast geometric ensembling [7], TreeNet [25], etc .

Original: In contrast, this work aims at exploring a combinatorial generalization of deep ensemble to obtain new features and improve the performance of uncertainty estimation.
Revised: In contrast, this work aims at exploring a combinatorial generalization of deep ensemble to obtain new features and improve the performance of uncertainty estimation.

Original: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable to stochastic weight averaging (SWA) [19].
Revised: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable to stochastic weight averaging (SWA) [16].

Original: 1.2 Contributions
Revised: 1.2 Contributions

Original: The main contributions of this paper are the following:
Revised: The main contributions of this paper are the following:

Original: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.
Revised: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.

Original: • We explore DCA at different levels of granularity and propose deep combinatorial weight averaging (DCWA) for ﬁne-grain DCA models.
Revised: • We explore DCA at different levels of granularity and propose deep combinatorial weight averaging (DCWA) for ﬁne-grain DCA models.

Original: It produces a new average model that improves on standard training and is competitive w.r.t.
Revised: It produces a new average model that improves on standard training and is competitive w.r.t.

Original: alternatives like stochastic weight averaging (SWA) [19].
Revised: alternatives like stochastic weight averaging (SWA) [16].

Original: • We introduce a consistency enforcing loss adapted for DCA training.
Revised: • We introduce a consistency enforcing loss adapted for DCA training.

Original: It strengthens the predictive consistency of DCA model proposals and consistently improves the performance of DCA and DCWA models.
Revised: It strengthens the predictive consistency of DCA model proposals and consistently improves the performance of DCA and DCWA models.

Original: • We conduct experiments on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCA for uncertainty-aware learning.
Revised: • We conduct experiments on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCA for uncertainty-aware learning.

Original:  Deep combinatorial aggregation (DCA)
Revised:  Deep combinatorial aggregation (DCA)

Original: In this section, we introduce the deep combinatorial aggregation (DCA) method.
Revised: In this section, we introduce the deep combinatorial aggregation (DCA) method.

Original: To simplify our discussion, we start with a layerwise setting and assume that our base model is a neural network with
Revised: To simplify our discussion, we start with a layerwise setting and assume that our base model is a neural network with

Original: L layers.
Revised: L layers.

Original: This setting can easily be generalized to other DCA variants discussed in Section 2.3.
Revised: This setting can easily be generalized to other DCA variants discussed in Section 2.3.

Original: [Figure]
Revised: [Figure]

Original: 2.1 Methodology
Revised: 2.1 Methodology

Original: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble [26] creates multiple copies of the entire model with different initializations, layerwise DCA instead creates multiple instances for each layer in the model.
Revised: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble [23] creates multiple copies of the entire model with different initializations, layerwise DCA instead creates multiple instances for each layer in the model.

Original: Randomly selected instances from network layers can be combined to form a variety of model proposals.
Revised: Randomly selected instances from network layers can be combined to form a variety of model proposals.

Original: As shown in Figure 1, this results in an exponential number of proposals w.r.t.
Revised: As shown in Figure 1, this results in an exponential number of proposals w.r.t.

Original: the network depth, since n sets of layer instances lead to n L total network proposals.
Revised: the network depth, since n sets of layer instances lead to n L total network proposals.

Original: To ensure consistency among the instance combinations, all layer instances are jointly trained: during each feed-forward, a random instance choice from each layer is made to construct a model proposal, and the parameters belonging to the selected layer instances are then updated via backpropagation.
Revised: To ensure consistency among the instance combinations, all layer instances are jointly trained: during each feed-forward, a random instance choice from each layer is made to construct a model proposal, and the parameters belonging to the selected layer instances are then updated via backpropagation.

Original: This differs from deep ensemble [26] where model copies are independently trained.
Revised: This differs from deep ensemble [23] where model copies are independently trained.

Original: In Appendix A we provide a pseudo-code for layerwise DCA training.
Revised: In Appendix A we provide a pseudo-code for layerwise DCA training.

Original: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCA model proposals and aggregate their predictions.
Revised: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCA model proposals and aggregate their predictions.

Original: 2.2 Understanding DCA
Revised: 2.2 Understanding DCA

Original: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoretical analysis to understand the assumptions it implicitly makes and their implications.
Revised: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoretical analysis to understand the assumptions it implicitly makes and their implications.

Original: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.
Revised: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.

Original: This translates to the assumption that the weight posterior p ( θ | y, x ) should be layerwise decomposable:
Revised: This translates to the assumption that the weight posterior p ( θ | y, x ) should be layerwise decomposable:

Original: [Equation]
Revised: [Equation]

Original: Is this a valid assumption?
Revised: Is this a valid assumption?

Original: Not exactly.
Revised: Not exactly.

Original: Admittedly, the posterior is proportional to the product of prior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) often satisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to
Revised: Admittedly, the posterior is proportional to the product of prior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) often satisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to

Original: Gaussian prior with constant diagonal covariance matrix.
Revised: Gaussian prior with constant diagonal covariance matrix.

Original: However, the likelihood term p ( y | x, θ ) is in general not decomposable.
Revised: However, the likelihood term p ( y | x, θ ) is in general not decomposable.

Original: In fact, following the directed graphical model [24] interpretation of standard neural networks [39], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ ) where h l represents hidden neurons in layer l .
Revised: In fact, following the directed graphical model [21] interpretation of standard neural networks [36], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ ) where h l represents hidden neurons in layer l .

Original: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwise decomposable itself
Revised: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwise decomposable itself

Original: [Equation]
Revised: [Equation]

Original: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h 1: L −
Revised: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h 1: L −

Original: [Equation]
Revised: [Equation]

Original: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.
Revised: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.

Original: Thus approximations are made when we perform layerwise DCA, and experiments show that this indeed results in some performance penalty.
Revised: Thus approximations are made when we perform layerwise DCA, and experiments show that this indeed results in some performance penalty.

Original: The above analysis also implies that weight aggregation at a coarser multilayer level could alleviate the issue.
Revised: The above analysis also implies that weight aggregation at a coarser multilayer level could alleviate the issue.

Original: This is also conﬁrmed by our empirical ﬁndings ( cf .
Revised: This is also conﬁrmed by our empirical ﬁndings ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: 2.3 Granularity of aggregation
Revised: 2.3 Granularity of aggregation

Original: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregation at different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise, multilayer, or modelwise levels.
Revised: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregation at different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise, multilayer, or modelwise levels.

Original: For convolutional neural networks, due to weight-sharing along the spatial dimensions, DCA can work with channel components instead of neurons.
Revised: For convolutional neural networks, due to weight-sharing along the spatial dimensions, DCA can work with channel components instead of neurons.

Original: Note that modelwise
Revised: Note that modelwise

Original: DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained with distinct subsets of training data which results from the random component selection, and it can beneﬁt from consistency enforcing loss introduced in Section 4.
Revised: DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained with distinct subsets of training data which results from the random component selection, and it can beneﬁt from consistency enforcing loss introduced in Section 4.

Original: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more varied predictions.
Revised: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more varied predictions.

Original: However, the DCA components are more tightly coupled, deviating more signiﬁcantly from the assumption of decomposable posterior, which could lead to worse performance.
Revised: However, the DCA components are more tightly coupled, deviating more signiﬁcantly from the assumption of decomposable posterior, which could lead to worse performance.

Original: This results in a tradeoff between performance and prediction variety.
Revised: This results in a tradeoff between performance and prediction variety.

Original: Alternatively, one can also enrich the set of model proposals by using more DCA instances, at the cost of a higher computational budget.
Revised: Alternatively, one can also enrich the set of model proposals by using more DCA instances, at the cost of a higher computational budget.

Original: This can also lead to improved performance ( cf .
Revised: This can also lead to improved performance ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise
Revised: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise

Original: DCA and multilayered DCA.
Revised: DCA and multilayered DCA.

Original: This comes from the fact that component instances from multilayered
Revised: This comes from the fact that component instances from multilayered

Original: DCA variants can have similar behaviors but dissimilar weights.
Revised: DCA variants can have similar behaviors but dissimilar weights.

Original: Neural networks admit a large number of equivalent reparameterizations via the reordering of hidden layer neurons.
Revised: Neural networks admit a large number of equivalent reparameterizations via the reordering of hidden layer neurons.

Original: While the joint training of DCA enforces consistency among different DCA components of the network model, it does not prevent equivalent reordering of hidden neurons inside multilayer components.
Revised: While the joint training of DCA enforces consistency among different DCA components of the network model, it does not prevent equivalent reordering of hidden neurons inside multilayer components.

Original: This issue does not occur for (sub)layerwise DCA cases.
Revised: This issue does not occur for (sub)layerwise DCA cases.

Original: To make the distinctions, we refer to (sub)layerwise cases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.
Revised: To make the distinctions, we refer to (sub)layerwise cases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.

Original:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation
Revised:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation

Original: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leads to an improved parameterization of the base model.
Revised: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leads to an improved parameterization of the base model.

Original: We discuss this procedure here in detail.
Revised: We discuss this procedure here in detail.

Original: Deep
Revised: Deep

Original: combinatorial weight averaging Since component instances of a ﬁne-grain DCA model have compatible weights after the joint training, it is sensible to consider their mean value.
Revised: combinatorial weight averaging Since component instances of a ﬁne-grain DCA model have compatible weights after the joint training, it is sensible to consider their mean value.

Original: This produces a new average parameterization for the base network model.
Revised: This produces a new average parameterization for the base network model.

Original: We refer to this process as deep combinatorial weight averaging (DCWA) .
Revised: We refer to this process as deep combinatorial weight averaging (DCWA) .

Original: Consider as an example the layerwise DCA model for a base neural network with l layers.
Revised: Consider as an example the layerwise DCA model for a base neural network with l layers.

Original: After the joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . . .
Revised: After the joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . . .

Original: , θ L 1: n ) , DCWA simply computes the average parameterization ¯ θ =
Revised: , θ L 1: n ) , DCWA simply computes the average parameterization ¯ θ =

Original: (¯ θ 1 , . .
Revised: (¯ θ 1 , . .

Original: , ¯ θ L ) for the base network model, where for each layer l , we have ¯ θ l = 1 n (cid:80) ni
Revised: , ¯ θ L ) for the base network model, where for each layer l , we have ¯ θ l = 1 n (cid:80) ni

Original: =1 θ li .
Revised: =1 θ li .

Original: Experiments show that DCWA achieves comparable test accuracy w.r.t.
Revised: Experiments show that DCWA achieves comparable test accuracy w.r.t.

Original: corresponding DCA predictions ( cf .
Revised: corresponding DCA predictions ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: It also consistently outperforms the standard training of the base network, and delivers comparable results w.r.t.
Revised: It also consistently outperforms the standard training of the base network, and delivers comparable results w.r.t.

Original: to SWA [19] ( cf .
Revised: to SWA [16] ( cf .

Original: Section 5.1).
Revised: Section 5.1).

Original: Comparison to SWA It is interesting to compare DCWA with SWA [19] since they are both weight averaging schemes that improve the standard training.
Revised: Comparison to SWA It is interesting to compare DCWA with SWA [16] since they are both weight averaging schemes that improve the standard training.

Original: This said, DCWA and SWA are based on different principles: SWA averages over the SGD trajectory while DCA relies on combining component aggregations.
Revised: This said, DCWA and SWA are based on different principles: SWA averages over the SGD trajectory while DCA relies on combining component aggregations.

Original: In practice, SWA requires custom learning rate scheduling and careful choice of end learning rate.
Revised: In practice, SWA requires custom learning rate scheduling and careful choice of end learning rate.

Original: Also, SWA requires an additional Batch Normalization update [19] to produce good predictions, which leads to an extra overhead.
Revised: Also, SWA requires an additional Batch Normalization update [16] to produce good predictions, which leads to an extra overhead.

Original: In contrast, DCWA does not have any of these issues and is simple to implement and deploy.
Revised: In contrast, DCWA does not have any of these issues and is simple to implement and deploy.

Original:  Consistency enforcing loss for DCA and DCWA
Revised:  Consistency enforcing loss for DCA and DCWA

Original: Through component combination, DCA is able to produce a combinatorial amount of model proposals.
Revised: Through component combination, DCA is able to produce a combinatorial amount of model proposals.

Original: However, during the joint training, each DCA component receives gradient updates from different model proposals.
Revised: However, during the joint training, each DCA component receives gradient updates from different model proposals.

Original: These updates can be inconsistent, which can lead to suboptimal training of the DCA model.
Revised: These updates can be inconsistent, which can lead to suboptimal training of the DCA model.

Original: To remedy this issue, we propose in this section a consistency enforcing loss to encourage consistency among DCA model proposals.
Revised: To remedy this issue, we propose in this section a consistency enforcing loss to encourage consistency among DCA model proposals.

Original: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage
Revised: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage

Original: DCA predictions agree with both the ground-truth and the predictions from other model proposals.
Revised: DCA predictions agree with both the ground-truth and the predictions from other model proposals.

Original: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ , instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )
Revised: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ , instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )

Original: = − log p ( y | x ; ˆ θ ) , the consistency enforcing loss includes an additional KL divergence term between the predictive output probability p ( y | x ; ˆ θ ) and a reference output probability ˜ p :
Revised: = − log p ( y | x ; ˆ θ ) , the consistency enforcing loss includes an additional KL divergence term between the predictive output probability p ( y | x ; ˆ θ ) and a reference output probability ˜ p :

Original: [Equation]
Revised: [Equation]

Original: ; ˆ θ ) .
Revised: ; ˆ θ ) .

Original: Here the reference output probability ˜ p should be chosen to reﬂect the predictions from other model proposals.
Revised: Here the reference output probability ˜ p should be chosen to reﬂect the predictions from other model proposals.

Original: In practice, we perform multiple feed-forward steps on the same input x using randomly chosen model proposals ( ˆ θ ( i ) )
Revised: In practice, we perform multiple feed-forward steps on the same input x using randomly chosen model proposals ( ˆ θ ( i ) )

Original: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simply uses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) =
Revised: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simply uses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) =

Original: (cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .
Revised: (cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .

Original: Note that the output predictions are reused and one additional feed-forward pass is needed to compute a ﬁrst reference distribution p (0) .
Revised: Note that the output predictions are reused and one additional feed-forward pass is needed to compute a ﬁrst reference distribution p (0) .

Original: The gradient updates from the multiple feed-forward steps can be accumulated together for a single optimization step on the global DCA parameter set Θ .
Revised: The gradient updates from the multiple feed-forward steps can be accumulated together for a single optimization step on the global DCA parameter set Θ .

Original: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) to regularize the predictive output distribution p .
Revised: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) to regularize the predictive output distribution p .

Original: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.
Revised: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.

Original: This is helpful for the output predictive distribution p to ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially in cases where they strongly disagree with each other.
Revised: This is helpful for the output predictive distribution p to ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially in cases where they strongly disagree with each other.

Original: Relation to existing approaches KL divergence as a regularization term in loss function appears in many contexts: for Bayesian neural networks [10, 4] it is used to regularize weight posterior; in variational autoencoders (VAE) [22] it regularizes the latent representation.
Revised: Relation to existing approaches KL divergence as a regularization term in loss function appears in many contexts: for Bayesian neural networks [8, 2] it is used to regularize weight posterior; in variational autoencoders (VAE) [19] it regularizes the latent representation.

Original: Applying KL divergence on output distributions has been proposed by Zhang et al.
Revised: Applying KL divergence on output distributions has been proposed by Zhang et al.

Original: [51] for knowledge distillation, and Song and Chai [42] for collaborative learning.
Revised: [46] for knowledge distillation, and Song and Chai [38] for collaborative learning.

Original: However, to the best of our knowledge, this has not been extensively explored in the context of uncertainty-aware learning.
Revised: However, to the best of our knowledge, this has not been extensively explored in the context of uncertainty-aware learning.

Original: Also, existing approaches [51, 42] require output predictions from all other models, which is impractical for DCA with a combinatorial amount of model proposals.
Revised: Also, existing approaches [46, 38] require output predictions from all other models, which is impractical for DCA with a combinatorial amount of model proposals.

Original: In comparison, our proposed consistency enforcing loss is efﬁcient and synergizes well with DCA training.
Revised: In comparison, our proposed consistency enforcing loss is efﬁcient and synergizes well with DCA training.

Original: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions.
Revised: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions.

Original: Since model diversity has been found beneﬁcial for ensemble approaches [47, 49, 32], the consistency enforcing loss has a mixed effect on uncertainty estimation and could be advantageous in some cases.
Revised: Since model diversity has been found beneﬁcial for ensemble approaches [43, 45, 29], the consistency enforcing loss has a mixed effect on uncertainty estimation and could be advantageous in some cases.

Original: In practice ( cf .
Revised: In practice ( cf .

Original: Section 5.3), we observe that the consistency enforcing loss boosts the predictive performance of
Revised: Section 5.3), we observe that the consistency enforcing loss boosts the predictive performance of

Original: DCWA models and improves the uncertainty estimation of modelwise DCA.
Revised: DCWA models and improves the uncertainty estimation of modelwise DCA.

Original: To further analyze the consistency enforcing loss, we empirically study its effect on the classiﬁcation performance of individual DCA model proposals.
Revised: To further analyze the consistency enforcing loss, we empirically study its effect on the classiﬁcation performance of individual DCA model proposals.

Original: The results are collected in Appendix C. We observe that the consistency enforcing loss improves the prediction results of individual models but reduces their diversity.
Revised: The results are collected in Appendix C. We observe that the consistency enforcing loss improves the prediction results of individual models but reduces their diversity.

Original:  Experiments
Revised:  Experiments

Original: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conduct a series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 and evaluate the uncertainty estimation of DCA models in Section 5.2.
Revised: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conduct a series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 and evaluate the uncertainty estimation of DCA models in Section 5.2.

Original: Also, ablation studies on DCA variants are included in Section 5.3 to further analyze several design aspects.
Revised: Also, ablation studies on DCA variants are included in Section 5.3 to further analyze several design aspects.

Original: For all experiments we use a common preactivation ResNet-20 architecture [13] as the base network, and ﬁve independent runs are executed to produce the ﬁnal results.
Revised: For all experiments we use a common preactivation ResNet-20 architecture [11] as the base network, and ﬁve independent runs are executed to produce the ﬁnal results.

Original: Our implementation uses PyTorch [35] and can run on a single modern GPU with 10Gb VRAM.
Revised: Our implementation uses PyTorch [32] and can run on a single modern GPU with 10Gb VRAM.

Original: 5.1 Image classiﬁcation using DCWA
Revised: 5.1 Image classiﬁcation using DCWA

Original: [Table]
Revised: [Table]

Original: Firstly, we evaluate the performance of the DCWA approach.
Revised: Firstly, we evaluate the performance of the DCWA approach.

Original: For this we use CIFAR-10 [25] and
Revised: For this we use CIFAR-10 [22] and

Original: SVHN [34] datasets and compare against the standard training and SWA [19] as baselines.
Revised: SVHN [31] datasets and compare against the standard training and SWA [16] as baselines.

Original: DCWA models are obtained from training layerwise DCA models with ﬁve copies and consistency enforcing loss is used.
Revised: DCWA models are obtained from training layerwise DCA models with ﬁve copies and consistency enforcing loss is used.

Original: SWA models are trained following the settings from Izmailov et al.
Revised: SWA models are trained following the settings from Izmailov et al.

Original: Test accuracy and negative log-likelihood (NLL) on both datasets are reported in Table 1.
Revised: Test accuracy and negative log-likelihood (NLL) on both datasets are reported in Table 1.

Original: We observe that DCWA and SWA have comparable results, and both of them outperform the standard training baseline.
Revised: We observe that DCWA and SWA have comparable results, and both of them outperform the standard training baseline.

Original: However, DCWA has the appealing practical advantages over SWA of not requiring custom learning rate scheduling nor readjustment of batch normalization layers.
Revised: However, DCWA has the appealing practical advantages over SWA of not requiring custom learning rate scheduling nor readjustment of batch normalization layers.

Original: 5.2 DCA for uncertainty-aware learning
Revised: 5.2 DCA for uncertainty-aware learning

Original: We now focus on comparing the uncertainty estimation of DCA models against a selection of practical uncertainty-aware learning methods, including MC dropout [8], SWAG [31], and deep ensemble [26].
Revised: We now focus on comparing the uncertainty estimation of DCA models against a selection of practical uncertainty-aware learning methods, including MC dropout [6], SWAG [28], and deep ensemble [23].

Original: Also, results from the standard training of the base network are reported for reference.
Revised: Also, results from the standard training of the base network are reported for reference.

Original: These baselines are compared with trunkwise 2 and modelwise DCA models.
Revised: These baselines are compared with trunkwise 2 and modelwise DCA models.

Original: Following the ablation study results in Table 3, consistency enforcing loss is used with modelwise DCA models while standard negative log-likelihood loss is used to train trunkwise DCA models for best performance.
Revised: Following the ablation study results in Table 3, consistency enforcing loss is used with modelwise DCA models while standard negative log-likelihood loss is used to train trunkwise DCA models for best performance.

Original: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain, distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.
Revised: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain, distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.

Original: We consider common metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score, and expected calibration error (ECE) [11].
Revised: We consider common metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score, and expected calibration error (ECE) [9].

Original: For OOD experiments, we follow the settings of Liang et al.
Revised: For OOD experiments, we follow the settings of Liang et al.

Original: [30] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,
Revised: [27] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,

Original: AUPR-in, AUPR-out, and AUROC in Appendix B.3.
Revised: AUPR-in, AUPR-out, and AUROC in Appendix B.3.

Original: Overall, we observe that modelwise DCA produces the best results both in terms of predictive performance and uncertainty estimation.
Revised: Overall, we observe that modelwise DCA produces the best results both in terms of predictive performance and uncertainty estimation.

Original: It is the method of choice if the performance of uncertaintyaware learning is the only concern.
Revised: It is the method of choice if the performance of uncertaintyaware learning is the only concern.

Original: Trunkwise DCA is overall competitive with deep ensemble and moreover it produces more diverse model proposals.
Revised: Trunkwise DCA is overall competitive with deep ensemble and moreover it produces more diverse model proposals.

Original: It offers a trade-off for more varied predictions.
Revised: It offers a trade-off for more varied predictions.

Original: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [25] and SVHN [34] datasets.
Revised: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [22] and SVHN [31] datasets.

Original: For all training we use SGD with momentum 0 .
Revised: For all training we use SGD with momentum 0 .

Original: We use a drop rate of 0 . 1 for MC dropout, and for SWAG we follow the settings from Maddox et al.
Revised: We use a drop rate of 0 . 1 for MC dropout, and for SWAG we follow the settings from Maddox et al.

Original: Deep ensemble is performed on ﬁve separate base networks.
Revised: Deep ensemble is performed on ﬁve separate base networks.

Original: And trunkwise and modelwise DCA models also use ﬁve copies.
Revised: And trunkwise and modelwise DCA models also use ﬁve copies.

Original: The results on CIFARare summarized in Table 2 and SVHN results are in Appendix B.1.
Revised: The results on CIFARare summarized in Table 2 and SVHN results are in Appendix B.1.

Original: [Table]
Revised: [Table]

Original: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracy and uncertainty estimation.
Revised: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracy and uncertainty estimation.

Original: Also, trunkwise DCA slightly outperforms deep ensemble in this case.
Revised: Also, trunkwise DCA slightly outperforms deep ensemble in this case.

Original: SVHN results in Appendix B.1 also conﬁrm these observations.
Revised: SVHN results in Appendix B.1 also conﬁrm these observations.

Original: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatch between the training data and the test set.
Revised: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatch between the training data and the test set.

Original: This calls for additional distributional shift experiments,
Revised: This calls for additional distributional shift experiments,

Original: [Figure]
Revised: [Figure]

Original: which gradually transform the in-domain test set to make it increasingly different than the training data
Revised: which gradually transform the in-domain test set to make it increasingly different than the training data

Original: In our case
Revised: In our case

Original: [Figure]
Revised: [Figure]

Original: we use the CIFAR-10-C [14] dataset
Revised: we use the CIFAR-10-C [12] dataset

Original: [Figure]
Revised: [Figure]

Original: which applies a wide range of corruptions to the original CIFAR-10 test set.
Revised: which applies a wide range of corruptions to the original CIFAR-10 test set.

Original: CIFAR-10-C quantizes corruptions into ﬁve severity levels where higher levels generate stronger corruptions and are more dissimilar to the in-domain test set.
Revised: CIFAR-10-C quantizes corruptions into ﬁve severity levels where higher levels generate stronger corruptions and are more dissimilar to the in-domain test set.

Original: We use the models previously trained on CIFAR-10 in-domain training data
Revised: We use the models previously trained on CIFAR-10 in-domain training data

Original: [Figure]
Revised: [Figure]

Original: and evaluate their performance on CIFAR-10-C separately for each severity level.
Revised: and evaluate their performance on CIFAR-10-C separately for each severity level.

Original: Figure 2 summarizes the results.
Revised: Figure 2 summarizes the results.

Original: Similar to the in-domain scenario
Revised: Similar to the in-domain scenario

Original: [Figure]
Revised: [Figure]

Original: modelwise DCA has the best results overall.
Revised: modelwise DCA has the best results overall.

Original: Trunkwise DCA has comparable test accuracies w.r
Revised: Trunkwise DCA has comparable test accuracies w.r

Original: [Figure]
Revised: [Figure]

Original: t. deep ensemble.
Revised: t. deep ensemble.

Original: It tends to outperform deep ensemble in terms of negative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by the additional plots in Appendix B.
Revised: It tends to outperform deep ensemble in terms of negative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by the additional plots in Appendix B.

Original: [Figure]
Revised: [Figure]

Original: Modelwise DCA shows the best performance for
Revised: Modelwise DCA shows the best performance for

Original: OOD detection, followed by trunkwise DCA.
Revised: OOD detection, followed by trunkwise DCA.

Original: Out-of-distribution results Additionally, we complement the distributional shift experiments with out-of-distribution image detection [15].
Revised: Out-of-distribution results Additionally, we complement the distributional shift experiments with out-of-distribution image detection [13].

Original: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.
Revised: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.

Original: Thus we have a binary classiﬁcation setting.
Revised: Thus we have a binary classiﬁcation setting.

Original: For our case, we reuse again the models trained on in-domain CIFAR-10 training data.
Revised: For our case, we reuse again the models trained on in-domain CIFAR-10 training data.

Original: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while test samples from SVHN are treated as negative outliers.
Revised: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while test samples from SVHN are treated as negative outliers.

Original: We show the ROC curves for DCA models and their baselines in Figure 3.
Revised: We show the ROC curves for DCA models and their baselines in Figure 3.

Original: Quantitative metrics following Liang et al.
Revised: Quantitative metrics following Liang et al.

Original: [30], which include
Revised: [27], which include

Original: FPR95, detection error, AUPR-in, AUPR-out and AUROC, are reported in Appendix B.3.
Revised: FPR95, detection error, AUPR-in, AUPR-out and AUROC, are reported in Appendix B.3.

Original: Again, we observe that modelwise DCA achieves the best OOD performance.
Revised: Again, we observe that modelwise DCA achieves the best OOD performance.

Original: Trunkwise DCA has the second-best performance and outperforms deep ensemble and SWAG.
Revised: Trunkwise DCA has the second-best performance and outperforms deep ensemble and SWAG.

Original: Inter- estingly, MC dropout turns out to perform poorly in the OOD case and is even worse than the standard training baseline.
Revised: Inter- estingly, MC dropout turns out to perform poorly in the OOD case and is even worse than the standard training baseline.

Original: The quantitative results in Appendix B.3 agree with the ROC curve.
Revised: The quantitative results in Appendix B.3 agree with the ROC curve.

Original: 5.3 Ablation studies
Revised: 5.3 Ablation studies

Original: We wrap up the experiment section with a series of ablation studies on variants of DCA models.
Revised: We wrap up the experiment section with a series of ablation studies on variants of DCA models.

Original: We consider three main aspects: the effect of granularity, the usage of consistency enforcing loss during training, and the inﬂuence of component instance count on DCA predictive results.
Revised: We consider three main aspects: the effect of granularity, the usage of consistency enforcing loss during training, and the inﬂuence of component instance count on DCA predictive results.

Original: We use the in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.
Revised: We use the in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.

Original: Effect of granularity To understand the effect of DCA component granularity on the performance of DCA models, we deﬁne ﬁve DCA model variants with different component granularities.
Revised: Effect of granularity To understand the effect of DCA component granularity on the performance of DCA models, we deﬁne ﬁve DCA model variants with different component granularities.

Original: This includes DCA models whose components are channels, layers, residual blocks, residual trunks, and entire models.
Revised: This includes DCA models whose components are channels, layers, residual blocks, residual trunks, and entire models.

Original: All DCA model variants are constructed with ﬁve sets of component instances and trained with the consistency enforcing loss (NLL loss leads to similar observations).
Revised: All DCA model variants are constructed with ﬁve sets of component instances and trained with the consistency enforcing loss (NLL loss leads to similar observations).

Original: We summarize the results of these DCA variants and their DCWA models in Figure 4.
Revised: We summarize the results of these DCA variants and their DCWA models in Figure 4.

Original: [Figure]
Revised: [Figure]

Original: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.
Revised: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.

Original: The metrics are all plotted in log scale.
Revised: The metrics are all plotted in log scale.

Original: Several interesting observations can be drawn from this experiment: For DCA models, coarser component granularity generally leads to better results.
Revised: Several interesting observations can be drawn from this experiment: For DCA models, coarser component granularity generally leads to better results.

Original: This agrees with the analysis in Section 2.2.
Revised: This agrees with the analysis in Section 2.2.

Original: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component
Revised: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component

Original: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWA variants improve on the standard baseline, while coarse-grain DCWA variants result in considerably worse performance than the standard baseline.
Revised: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWA variants improve on the standard baseline, while coarse-grain DCWA variants result in considerably worse performance than the standard baseline.

Original: This also matches our analysis in Section 2.3.
Revised: This also matches our analysis in Section 2.3.

Original: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better than random ( i.e . 10%) accuracy.
Revised: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better than random ( i.e . 10%) accuracy.

Original: In this case, reparameterization mismatch only happens on the side branches of the residual blocks, and the trained weights on the main branch are still able to produce meaningful predictions despite the erroneous parameterization on the residual side branches.
Revised: In this case, reparameterization mismatch only happens on the side branches of the residual blocks, and the trained weights on the main branch are still able to produce meaningful predictions despite the erroneous parameterization on the residual side branches.

Original: [Table]
Revised: [Table]

Original: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained with negative log-likelihood loss with those trained with consistency enforcing loss.
Revised: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained with negative log-likelihood loss with those trained with consistency enforcing loss.

Original: We summarize the results on CIFAR-10 in Table 3.
Revised: We summarize the results on CIFAR-10 in Table 3.

Original: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.
Revised: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.

Original: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of
Revised: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of

Original: NLL and ECE.
Revised: NLL and ECE.

Original: On the other hand, negative log-likelihood loss is the better choice for the trunkwise
Revised: On the other hand, negative log-likelihood loss is the better choice for the trunkwise

Original: DCA model.
Revised: DCA model.

Original: In Appendix B.4 we provide additional results for other DCA variants.
Revised: In Appendix B.4 we provide additional results for other DCA variants.

Original: [Figure]
Revised: [Figure]

Original: Number of component instances Finally, we investigate the inﬂuence of DCA instance count.
Revised: Number of component instances Finally, we investigate the inﬂuence of DCA instance count.

Original: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial to multiply the training epochs and per-minibatch gradient backpropagations accordingly.
Revised: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial to multiply the training epochs and per-minibatch gradient backpropagations accordingly.

Original: For example, in our experiments, we scheduleepochs for standard baseline training.
Revised: For example, in our experiments, we scheduleepochs for standard baseline training.

Original: Then, for training a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradient over three backpropagations on randomly selected DCA proposals for parameter update.
Revised: Then, for training a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradient over three backpropagations on randomly selected DCA proposals for parameter update.

Original: We train a series of layerwise DCA models with two, three, four, and ﬁve sets of component instances, and collect their results in Figure 5.
Revised: We train a series of layerwise DCA models with two, three, four, and ﬁve sets of component instances, and collect their results in Figure 5.

Original: In general, we observe that more DCA instances lead to better results (higher accuracy and lower
Revised: In general, we observe that more DCA instances lead to better results (higher accuracy and lower

Original: NLL loss in Figure 5).
Revised: NLL loss in Figure 5).

Original: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.
Revised: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.

Original: Moreover, we see that a minimal DCA model with two sets of components can already signiﬁcantly improve the performance compared to the standard base network training case.
Revised: Moreover, we see that a minimal DCA model with two sets of components can already signiﬁcantly improve the performance compared to the standard base network training case.

Original:  Discussion
Revised:  Discussion

Original: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that has also a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learning rate scheduling of SWAG, requires no elaborate hyperparameter search.
Revised: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that has also a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learning rate scheduling of SWAG, requires no elaborate hyperparameter search.

Original: We ﬁnd that coarse-grain
Revised: We ﬁnd that coarse-grain

Original: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms all other uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios; trunkwise DCA produces more diverse predictions while still delivering competitive performance w.r.t.
Revised: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms all other uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios; trunkwise DCA produces more diverse predictions while still delivering competitive performance w.r.t.

Original: deep ensemble, which is the best among existing baselines.
Revised: deep ensemble, which is the best among existing baselines.

Original: Additionally, we discover that ﬁne-grain DCA can produce a DCWA parameterization for the base network.
Revised: Additionally, we discover that ﬁne-grain DCA can produce a DCWA parameterization for the base network.

Original: DCWA improves the predictive performance compared to the standard baseline training.
Revised: DCWA improves the predictive performance compared to the standard baseline training.

Original: It has on-par performance with
Revised: It has on-par performance with

Original: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.
Revised: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.

Original: Thus
Revised: Thus

Original: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.
Revised: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.

Original: Similar to deep ensemble, DCA requires multiple times the computational budget to perform the joint training, compared to the standard baseline.
Revised: Similar to deep ensemble, DCA requires multiple times the computational budget to perform the joint training, compared to the standard baseline.

Original: While in this work we focused on the predictive power, we plan to improve the efﬁciency of DCA in future work.
Revised: While in this work we focused on the predictive power, we plan to improve the efﬁciency of DCA in future work.

Original: Apart from this, it can also be interesting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain
Revised: Apart from this, it can also be interesting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain

Original: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimations can only be attained via coarse-grain DCAs instead.
Revised: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimations can only be attained via coarse-grain DCAs instead.

Original: Additionally, it is possible to combine DCA with other uncertainty-aware methods, which might further improve the performance.
Revised: Additionally, it is possible to combine DCA with other uncertainty-aware methods, which might further improve the performance.

Original: We plan to further investigate these aspects in future work.
Revised: We plan to further investigate these aspects in future work.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC
Revised: This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC

Original: Advanced Grant SIMULACRON.
Revised: Advanced Grant SIMULACRON.

Original: The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers and area chair for their constructive feedback.Checklist
Revised: The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers and area chair for their constructive feedback.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]

Original: In Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Pseudo-code for DCA training

Original: Algorithm 1 depicts the pseudocode for training a layerwise DCA model.

Original: Inputs : Dataset D , neural network f with n sets of parameters Θ =

Original: ( θ 11: n , . .

Original: , θ L 1: n ) , with θ li the i -th parameter instance of layer l . sampleDCAProposal

Original: Draw random indices i 1 , . .

Original: , i L ∈

Original: ; return ˆ θ = ( θ 1 i 1 , . . .

Original: , θ Li

Original: ) ; 4 repeat foreach minibatch B ∈ D do ˆ θ ← sampleDCAProposal( Θ ); loss ← feedForward( f, ˆ θ , B ); ∆ ˆ θ ← loss.backward(); Θ ← updateDCAParameters( Θ , ∆ ˆ θ ); end 11 until end of training ;

Original: B Additional experimental results B.1 In-domain results on SVHN dataset

Original: Table 4 summarizes the results of in-domain image classiﬁcation on SVHN dataset.

Original: [Table]

Original: [Table]

Original: B.2 Additional plots for distributional shift experiment

Original: Figure 6 reports Brier scores and ECEs of the distributional shift experiments on CIFAR-10-C.

Original: B.3 Additional quantitative results for out-of-distribution experiment

Original: Table 5 provides quantitative metrics for the OOD detection task using CIFAR-10 as the in-domain dataset and SVHN as the OOD outliers.

Original: The following metrics are reported:

Original: False positive rate at 95% true positive rate (FPR@95%)

Original: A threshold metric which reports the false positive rate at the threshold where the true positive rate is 95%; Detection error The minimum of average values of false positive rate and false negative rate evaluated at different decision boundaries d : min d ∈ [0 , 1] ( FPR | d + FNR | d ) / 2 ; Area under the receiver operating characteristic curve (AUROC)

Original: This metric reports the area under the receiver operating characteristic (ROC) curve;

Original: [Figure]

Original: [Table]

Original: Area under the precision–recall curve (AUPR-in / AUPR-out)

Original: This metric measures the area under the precision–recall (AUPR) curve, AUPR-in reports the AUPR where the in-domain data are taken as positive samples, whereas AUPR-out instead treats OOD data as positive samples and in-domain data as negative ones.

Original: B.4 Additional results for ablation study on consistency enforcing loss

Original: Table 6 displays additional results for channelwise, layerwise and blockwise DCA variants.

Original: C Performance of individual DCA model proposals

Original: To further analyze the effect of the consistency enforcing loss for DCA models, we inspect the predictions of the individual model copies from modelwise DCAs training with and without the consistency enforcing loss.

Original: The results are reported in Table 7.

Original: We observe that individual models trained with the consistency enforcing loss have better results compared to the NLL case, which shows that CEL improves the training of individual components.

Original: Compared to the baseline standard training, individual model proposals from trained DCA models

Original: [Table]

Original: [Table]

Original: have worse classiﬁcation performance, however their aggregation yields superior results both in terms of accuracy and uncertainty estimation.

Original: In Table

Original: [Table]

Original: we also see that results from individual models trained without CEL have larger standard deviations, which hints that the predictions might be more diverse in this case.

Original: To further conﬁrm this, we employ the following metrics to quantify the diversity of individual predictions [1] (assuming we have M individual models and C output classes, KL denotes the KL-divergence, Var denotes the variance, and for a given input x , p mc ( x ) denotes the predicted probability of class c from model m ):

Original: Pairwise KL divergence

Original: [Equation]

Original: Classwise variance

Original: [Equation]

Original: Jensen-Shannon divergence

Original: [Equation]

Original: The results are collected in Table 8 and conﬁrm that individual models trained without CEL have more diverse predictions.

Original: [Table]

Original: D Additional results using VGG and DenseNet

Original: In previous experiments we focus on the preactivation ResNet-20 base model.

Original: Since the DCA approach does not rely on speciﬁc choice of network architecture, we expect the conclusions to hold for other network structures.

Original: To empirically verify this

Original: [Table]

Original: we conduct additional in-domain experiments on CIFAR-10 using two more architectures: VGG-16 [40] and DenseNet-40 [18].

Original: The results are summarized in Table 9 for the VGG backbone and in Table 10 for the DenseNet case.

Original: We see that modelwise DCA consistently achieves the best results both in terms of accuracy and uncertainty estimation for all tested network architectures.

Original: [Table]

Original: [Table]

Original: E Text classiﬁcation experiments

Original: Additionally

Original: [Table]

Original: we consider the performance of DCA for natural language processing tasks.

Original: Especially, we address the task of text classiﬁcation on the AG News dataset [50] using a Seq2seq model with attention [2].

Original: We adapt from an existing implementation 3 and collect the results in Table 11.

Original: We observe that modelwise DCA achieves the best accuracy when trained with the NLL loss.

Original: Using the consistency enforcing loss, modelwise DCA achieves the best ECE.

Original: [Table]