Original: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION
Revised: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Ensembles (DE) are a prominent approach achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-ofdistribution detection.
Revised: Deep Ensembles (DE) are a prominent approach achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-ofdistribution detection.

Original: However, hardware limitations of real-world systems constrain to smaller ensembles and lower capacity networks, significantly deteriorating their performance and properties.
Revised: However, hardware limitations of real-world systems constrain to smaller ensembles and lower capacity networks, significantly deteriorating their performance and properties.

Original: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.
Revised: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.

Original: We leverage grouped convolutions to parallelize the ensemble into a single common backbone and forward pass to improve training and inference speeds.
Revised: We leverage grouped convolutions to parallelize the ensemble into a single common backbone and forward pass to improve training and inference speeds.

Original: PE is designed to work under the memory budget of a single standard neural network.
Revised: PE is designed to work under the memory budget of a single standard neural network.

Original: Through extensive studies we show that PE faithfully preserve the properties of DE, e.g., diversity, and match their performance in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.
Revised: Through extensive studies we show that PE faithfully preserve the properties of DE, e.g., diversity, and match their performance in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose particularly high reliability and performance requirements over a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.
Revised: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose particularly high reliability and performance requirements over a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.

Original: Although their performance across all key dimensions has been dramatically improved in the last years, vanilla
Revised: Although their performance across all key dimensions has been dramatically improved in the last years, vanilla

Original: Deep Neural Networks (DNNs) still have several shortcomings, notably the overconfidence on both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).
Revised: Deep Neural Networks (DNNs) still have several shortcomings, notably the overconfidence on both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).

Original: Deep
Revised: Deep

Original: Ensembles (DE) (Lakshminarayanan et al., 2017) arise as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.
Revised: Ensembles (DE) (Lakshminarayanan et al., 2017) arise as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.

Original: By averaging predictions or by voting, DE achieve high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.
Revised: By averaging predictions or by voting, DE achieve high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.

Original: Thanks to the simplicity and effec- tiveness of the ensembling strategy (Dietterich, 2000), DE have become largely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: Thanks to the simplicity and effec- tiveness of the ensembling strategy (Dietterich, 2000), DE have become largely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: DE tick all boxes in the requirements list for real-world applications, except for computational efficiency.
Revised: DE tick all boxes in the requirements list for real-world applications, except for computational efficiency.

Original: In fact, DE are in practice computationally demanding (required memory storage, number of operations, inference time) both for training and testing as their costs grow linearly with the size of the ensembles.
Revised: In fact, DE are in practice computationally demanding (required memory storage, number of operations, inference time) both for training and testing as their costs grow linearly with the size of the ensembles.

Original: Their computational costs are therefore prohibitive under tight hardware constraints.
Revised: Their computational costs are therefore prohibitive under tight hardware constraints.

Original: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015), architectures with ensemble-imitating layers (Wen et al., 2019; Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016; Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018), efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).
Revised: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020), architectures with ensemble-imitating layers (Wen et al., 2019; Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016; Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018), efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).

Original: These approaches typically improve storage usage, train cost or inference time at the cost of lower accuracy and lower diversity in the predictions.
Revised: These approaches typically improve storage usage, train cost or inference time at the cost of lower accuracy and lower diversity in the predictions.

Original: An essential property of ensembles that enables improved predictive uncertainty estimation is related to the diversity of predictions in the ensemble.
Revised: An essential property of ensembles that enables improved predictive uncertainty estimation is related to the diversity of predictions in the ensemble.

Original: Perrone & Cooper (1992) have shown that the independence of individuals is critical to the success of ensembling.
Revised: Perrone & Cooper (1992) have shown that the independence of individuals is critical to the success of ensembling.

Original: Fort et al.
Revised: Fort et al.

Original: (2019) argue that the diversity of DE due to randomness from weight initialization, data augmentation and batching, stochastic gradient updates, etc., is superior to various other efficient ensembling approaches, in spite of their predictive performance boosts.
Revised: (2019) argue that the diversity of DE due to randomness from weight initialization, data augmentation and batching, stochastic gradient updates, etc., is superior to various other efficient ensembling approaches, in spite of their predictive performance boosts.

Original: Few works manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes, image throughput, etc.).
Revised: Few works manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes, image throughput, etc.).

Original: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular having a set of independent networks, in a computationally efficient manner.
Revised: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular having a set of independent networks, in a computationally efficient manner.

Original: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.
Revised: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.

Original: We build upon this idea and devise a strategy based on small networks towards matching the performance of an ensemble of large networks.
Revised: We build upon this idea and devise a strategy based on small networks towards matching the performance of an ensemble of large networks.

Original: To this end we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.
Revised: To this end we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.

Original: The parameters of each subnetworks are not shared across subnetworks, leading to independent smaller models.
Revised: The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models.

Original: This enables fast training and inference times, while predictive uncertainty quantification is close to DE (Figure 1).
Revised: This enables fast training and inference times, while predictive uncertainty quantification is close to DE (Figure 1).

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured-sparsity for Deep Ensembles; • We extensively evaluate PE in terms of accuracy, calibration, OOD detection, and dataset shift on classification and regression tasks.
Revised: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured-sparsity for Deep Ensembles; • We extensively evaluate PE in terms of accuracy, calibration, OOD detection, and dataset shift on classification and regression tasks.

Original: We demonstrate that PE achieves state-of-theart predictive uncertainty quantification.
Revised: We demonstrate that PE achieves state-of-theart predictive uncertainty quantification.

Original: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.
Revised: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we present the formalism for this work and offer a short background on grouped convolutions and ensembles of DNNs.
Revised: In this section, we present the formalism for this work and offer a short background on grouped convolutions and ensembles of DNNs.

Original: 2. B ACKGROUND ON CONVOLUTIONS
Revised: 2. B ACKGROUND ON CONVOLUTIONS

Original: The convolutional layer (LeCun et al., 1989) consists in a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C
Revised: The convolutional layer (LeCun et al., 1989) consists in a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C

Original: j +1 × C j × s 2 j with C j , H j , W j three integers that represent respectively the number of channels, the height and the width of h j .
Revised: j +1 × C j × s 2 j with C j , H j , W j three integers that represent respectively the number of channels, the height and the width of h j .

Original: C j +

Revised: C j +

Original: and s j are also two integers corresponding respectively to the number of channels of h j +1 (the output of the layer) and the kernel size.
Revised: and s j are also two integers corresponding respectively to the number of channels of h j +1 (the output of the layer) and the kernel size.

Original: Finally, j is the layer’s index and will be fixed in the following formulae.
Revised: Finally, j is the layer’s index and will be fixed in the following formulae.

Original: The main notations are detailed in Table 3, in the supplementary  material.
Revised: The main notations are detailed in Table 3, in the supplementary [Figure] material.

Original: The bias of convolution layers will be omitted in the following for simplicity.
Revised: The bias of convolution layers will be omitted in the following for simplicity.

Original: Hence the output value of the convolution layer, denoted ⊛ , is:
Revised: Hence the output value of the convolution layer, denoted ⊛ , is:

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ
Revised: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ

Original: ( z j ) with ϕ an activation function.
Revised: ( z j ) with ϕ an activation function.

Original: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in
Revised: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in

Original: ResNext (Xie et al., 2017) to train several DNN branches in parallel.
Revised: ResNext (Xie et al., 2017) to train several DNN branches in parallel.

Original: The grouped convolution operation with γ groups and weights ω iγ ∈ R C
Revised: The grouped convolution operation with γ groups and weights ω iγ ∈ R C

Original: j +1 ×
Revised: j +1 ×

Original: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.
Revised: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.

Original: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC j + (cid:107) , which only uses 1 γ of the input channels.
Revised: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC [Equation] (cid:107) , which only uses 1 γ of the input channels.

Original: [Equation]
Revised: [Equation]

Original: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×
Revised: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×

Original: C j × s 2 j such that mask jm ( k, l, : , :)
Revised: C j × s 2 j such that mask jm ( k, l, : , :)

Original: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)
Revised: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)

Original: =
Revised: =

Original: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .
Revised: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .

Original: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) 
Revised: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) m =

Original: [Equation]

Original:  j m and the grouped convolution can therefore be rewritten as:
Revised: 0 mask j m and the grouped convolution can therefore be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where ◦ is the Hadamard product.
Revised: where ◦ is the Hadamard product.

Original: 2. B ACKGROUND ON D EEP E NSEMBLES
Revised: 2. B ACKGROUND ON D EEP E NSEMBLES

Original: For a classification problem, let us define a dataset D
Revised: For a classification problem, let us define a dataset D

Original: = { x i , y i } |D| i =1 containing |D| pairs of samples x i ∈
Revised: = { x i , y i } |D| i =1 containing |D| pairs of samples x i ∈

Original: R H j × W j and one-hot-encoded labels
Revised: R H j × W j and one-hot-encoded labels

Original: y i ∈
Revised: y i ∈

Original: R N C modeled as the realisation of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.
Revised: R N C modeled as the realisation of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.

Original: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i
Revised: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i

Original: =
Revised: =

Original: f θ ( x i ) =
Revised: f θ ( x i ) =

Original: P ( Y = y
Revised: P ( Y = y

Original: i
Revised: i

Original: | X = x i ; θ ) .
Revised: | X = x i ; θ ) .

Original: This approach consists in considering the prediction ˆ y as parameters of a Multinoulli distribution.
Revised: This approach consists in considering the prediction ˆ y as parameters of a Multinoulli distribution.

Original: To improve the accuracy of predictions but also the accuracy of the predicted uncertainties and the detection of OOD samples, Lakshminarayanan et al.
Revised: To improve the accuracy of predictions but also the accuracy of the predicted uncertainties and the detection of OOD samples, Lakshminarayanan et al.

Original: (2017) have proposed to ensemble M randomly equivalent architectures 3,3x3,
Revised: (2017) have proposed to ensemble M randomly equivalent architectures 3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Original: 72,1x1,144groups =
Revised: 72,1x1,144groups =

Original: Rearrange xxx3,3x3,
Revised: Rearrange xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Original: 72,1x1,144groups =a) b) c) initialized deep Neural Networks as a large predictor called Deep Ensembles.
Revised: 72,1x1,144groups =a) b) c) initialized deep Neural Networks as a large predictor called Deep Ensembles.

Original: These ensembles can be seen as a discrete approximation of the intractable bayesian marginalization on the weights, according to Wilson & Izmailov (2020).
Revised: These ensembles can be seen as a discrete approximation of the intractable bayesian marginalization on the weights, according to Wilson & Izmailov (2020).

Original: If we note { θ m } M − 1 m =

Revised: If we note { θ m } M − 1 m =

Original: the set of trained weights for the M
Revised: the set of trained weights for the M

Original: DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (4).
Revised: DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (4).

Original: [Equation]
Revised: [Equation]

Original:  P ACKED -E NSEMBLES
Revised:  P ACKED -E NSEMBLES

Original: In this section, we present how to  train multiple subnetworksefficiently using grouped convolution.
Revised: In this section, we present how to efficiently train multiple subnetworks using grouped convolution.

Original: Then, we explain how our new architectures are equivalent to training several networks in parallel.
Revised: Then, we explain how our new architectures are equivalent to training several networks in parallel.

Original: 3. R EVISITING D EEP E NSEMBLES
Revised: 3. R EVISITING D EEP E NSEMBLES

Original: Although Deep Ensembles provide undisputed benefits, they also come with the major drawback that the training time and the memory usage in inference increase linearly with the number of networks.
Revised: Although Deep Ensembles provide undisputed benefits, they also come with the major drawback that the training time and the memory usage in inference increase linearly with the number of networks.

Original: To alleviate these problems, we propose assembling small sub-neural networks, which are DNNs with fewer parameters.
Revised: To alleviate these problems, we propose assembling small subnetworks, which are DNNs with fewer parameters.

Original: Moreover, while ensembles to this day have been trained sequentially, we propose to leverage grouped convolutions to accelerate massively their training and inference computations thanks to their smaller size.
Revised: Moreover, while ensembles to this day have been trained sequentially, we propose to leverage grouped convolutions to accelerate massively their training and inference computations thanks to their smaller size.

Original: The propagation of grouped convolutions with M groups, M being the number of subnetworks in the ensemble, ensures that the sub-neural networks are trained independently while dividing their encoding dimension by a factor M .
Revised: The propagation of grouped convolutions with M groups, M being the number of subnetworks in the ensemble, ensures that the latters are trained independently while dividing their encoding dimension by a factor M .

Original: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.
Revised: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.

Original: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as a expansion factor.
Revised: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as a expansion factor.

Original: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.
Revised: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.

Original: For an ensemble of M subnetworks,
Revised: For an ensemble of M subnetworks,

Original: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor αM and the inference of our ensemble is computed with the following formula:
Revised: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor αM and the inference of our ensemble is computed with the following formula:

Original: [Equation]
Revised: [Equation]

Original: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×
Revised: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×

Original: s 2 j .
Revised: s 2 j .

Original: In the following, we also add another hyperparameter γ which corresponds to the number of groups of each subnetworks of the Packed-Ensembles and creates another level of sparsity to our ensembles.
Revised: In the following, we also add another hyperparameter γ which corresponds to the number of groups of each subnetworks of the Packed-Ensembles and creates another level of sparsity to our ensembles.

Original: These groups are also called ”subgroups” and are applied to the different subnetworks.
Revised: These groups are also called ”subgroups” and are applied to the different subnetworks.

Original: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , where the hyperparameters are in the parentheses.
Revised: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , where the hyperparameters are in the parentheses.

Original: In this work, we consider the case of a constant number of subgroups across the layers and therefore γ divides αC
Revised: In this work, we consider the case of a constant number of subgroups across the layers and therefore γ divides αC

Original: j for all j . m=m=
Revised: j for all j . m=

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: m= m=
Revised: m= m=

Original: [Figure]
Revised: [Figure]

Original: 3. C OMPUTATIONAL COST
Revised: 3. C OMPUTATIONAL COST

Original: In a convolutional setting, the number of parameters in a layer involving C j input channels, C j + output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)
Revised: In a convolutional setting, the number of parameters in a layer involving C j input channels, C j + output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)

Original: αC j M αC
Revised: αC j M αC

Original: j +1 M s 2 j γ − 1 (cid:105) .
Revised: j +1 M s 2 j γ − 1 (cid:105) .

Original: The same formula applies to dense layers as 1 × 1 convolutions.
Revised: The same formula applies to dense layers as 1 × 1 convolutions.

Original: If architectures of the estimators are fully convolutional or dense, two specific cases emerge.
Revised: Whenever the architectures of the subnetworks are fully convolutional or dense, two specific cases emerge.

Original: If α = √ M : the number of parameters in the ensemble is equal to the number of parameters in a single model and if α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent  to Deep Ensembles).
Revised: If α = √ M , the number of parameters in the ensemble is equal to the number of parameters in a single model. With α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent in size to Deep Ensembles).

Original: 3. I MPLEMENTATION DETAILS
Revised: 3. I MPLEMENTATION DETAILS

Original: In this paper, we propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.
Revised: In this paper, we propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.

Original: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) on Figure 3, by the parallel implementations (b) and (c) .
Revised: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) on Figure 3, by the parallel implementations (b) and (c) .

Original: Figure 3 summarizes different equivalent architectures for a simple ensemble of
Revised: Figure 3 summarizes different equivalent architectures for a simple ensemble of

Original: M = 3 DNNs with 3 convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .
Revised: M = 3 DNNs with 3 convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .

Original: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the ”rearrange”operation).
Revised: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the ”rearrange”operation).

Original: This yields a feature map h j , of size M × C
Revised: This yields a feature map h j , of size M × C

Original: i × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.
Revised: i × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.

Original: One solution to keep the same batch size is to repeat the batch M times so that its size is equal to B after the rearrangement.
Revised: One solution to keep the same batch size is to repeat the batch M times so that its size is equal to B after the rearrangement.

Original: Using convolutions with M groups and γ subgroups per subnetworks, each feature map is convoluted separately by each subnetworks and yields its own independent output.
Revised: Using convolutions with M groups and γ subgroups per subnetworks, each feature map is convoluted separately by each subnetwork and yields its own independent output.

Original: Grouped convolutions are propagated until the end to ensure that gradients stay independent between sub-neural networks.
Revised: Grouped convolutions are propagated until the end to ensure that gradients stay independent between subnetworks.

Original: Other operations such as Batch Normalization (Ioffe & Szegedy, 2015) can be applied directly as long as they can be grouped or have independent actions on each channel.
Revised: Other operations such as Batch Normalization (Ioffe & Szegedy, 2015) can be applied directly as long as they can be grouped or have independent actions on each channel.

Original: Figure 4a illustrates the mask used to code Packed-Ensembles in the case where M = 2 .
Revised: Figure 4a illustrates the mask used to code PackedEnsembles in the case where M = 2 .

Revised: Similarly, Figure 4b shows the mask with M = 2 and γ = 2 .

Original: Finally, (b) and (c) are also equivalent.
Revised: Finally, (b) and (c) are also equivalent.

Original: It is indeed possible to replace the ”rearrange” operation and the first grouped convolution by a standard convolution if the same images are to be provided at the same time to all the subnetworks.
Revised: It is indeed possible to replace the ”rearrange” operation and the first grouped convolution by a standard convolution if the same images are to be provided at the same time to all the subnetworks.

Original: We confirm inthe Appendix F that this procedure is not detrimental to the performance of the ensemble and we take advantage from this property to provide this final optimization and simplification.
Revised: We confirm in Appendix F that this procedure is not detrimental to the performance of the ensemble and we take advantage from this property to provide this final optimization and simplification.

Original: Figure 4b shows the mask used to code Packed-Ensembles in the case where M = 2 and γ = 2 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To validate the performance of our method, we conduct experiments on classification tasks and we measure the effect of the parameters α and γ .
Revised: To validate the performance of our method, we conduct experiments on classification tasks and we measure the effect of the parameters α and γ .

Original: Regression tasks are detailed in Appendix N.
Revised: Regression tasks are detailed in Appendix N.

Original: 4. D ATASETS AND ARCHITECTURES
Revised: 4. D ATASETS AND ARCHITECTURES

Original: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to different data complexities.
Revised: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to different data complexities.

Original: As we propose to replace a single model architecture with several subnetworks, we study the behavior of Packed-Ensembles on architectures with various sizes: ResNet-18 and ResNet-50 (He et al., 2016), and Wide ResNet28-10 (Zagoruyko & Komodakis, 2016).We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and two other approximated ensembles from the literature:
Revised: As we propose to replace a single model architecture with several subnetworks, we study the behavior of Packed-Ensembles on architectures with various sizes: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet28-10 (Zagoruyko & Komodakis, 2016). We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and three other approximated ensembles from the literature:

Original: BatchEnsemble (Wen et al., 2019) and MIMO (Havasi et al., 2020).
Revised: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020) and Masksembles (Durasov et al., 2021).

Original: Secondly, we report our results for Packed-Ensembles on ImageNet, which we compare against Deep Ensembles.
Revised: Secondly, we report our results for Packed-Ensembles on ImageNet, which we compare against all baselines.

Original: We run experiments with Resnet-50 and Resnet-50x4.
Revised: We run experiments with ResNet-50 and ResNet-50x4.

Original: All training runs are started from scratch.
Revised: All training runs are started from scratch.

Original: 4.1. M ETRICS
Revised: 4.1. M ETRICS

Original: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and Negative Log-Likelihood (NLL).
Revised: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and Negative Log-Likelihood (NLL).

Original: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2 , and measure the quality of the
Revised: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2 , and measure the quality of the

Original: OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating
Revised: OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating

Original: Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in % similarly to Hendrycks & Gimpel (2017)
Revised: Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in % similarly to Hendrycks & Gimpel (2017)

Original: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).
Revised: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).

Original: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum in the output vector to be the predicted class of the ensemble.
Revised: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble.

Original: We define the prediction confidence as the value of this maximum (also called maximum softmax probability).
Revised: We define the prediction confidence as the value of this maximum (also called maximum softmax probability).

Original: For the out-of-distribution detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution (ID) data and OOD data using the maximum softmax probability as OOD criterion.
Revised: For the out-of-distribution detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution (ID) data and OOD data using the maximum softmax probability as OOD criterion.

Original: We discuss the different OOD criteria in appendix E.
Revised: We discuss the different OOD criteria in appendix E.

Original: For ImageNet, we use two out-of-distribution datasets: ImageNet-O (IO) (Hendrycks et al., 2021b) and Texture (T) (Wang et al., 2022) and use the Mutual Information ( MI ) as criterion for the ensembles techniques (see Appendix E for  MI ) and the maximum softmax probability for the single model.
Revised: For ImageNet, we use two out-of-distribution datasets: ImageNet-O (IO) (Hendrycks et al., 2021b) and Texture (T) (Wang et al., 2022), and use the Mutual Information ( MI ) as criterion for the ensembles techniques (see Appendix E for details on MI ) and the maximum softmax probability for the single model and MIMO.

Original: To measure the robustness under distribution shift, we use ImageNet-R (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denoted respectively rAcc, rECE, and rNLL on this dataset.
Revised: To measure the robustness under distribution shift, we use ImageNetR (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denoted respectively rAcc, rECE, and rNLL on this dataset.

Original: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.
Revised: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.

Original: Both are open-source Python frameworks.
Revised: Both are open-source Python frameworks.

Original: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.
Revised: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.

Original: Most trainings are completed on a single Nvidia
Revised: Most trainings are completed on a single Nvidia

Original: RTX 3090 except on ImageNet for which we use 4 to 8 Nvidia A100-80GB.
Revised: RTX 3090 except on ImageNet for which we use 2 to 8 Nvidia A100-80GB.

Original: 4.1. R ESULTS
Revised: 4.1. R ESULTS

Original: Table 1 presents the average performance over five runs on the classification task using the hyperparameters on Table 4.
Revised: Table 1 presents the average performance over five runs on the classification task using the hyperparameters on Table 4.

Original: We showcase that in the particular setting of α = 2 and γ = 2 , PackedEnsembles yields similar results to Deep Ensembles while having a lower memory cost than the single model.
Revised: We showcase that in the particular setting of α = 2 and γ = 2 , PackedEnsembles yields similar results to Deep Ensembles while having a lower memory cost than the single model.

Original: On CIFAR-10 (Krizhevsky, 2009), we notice that its relative performance compared to Deep Ensembles seems to increase as the original architecture gets bigger.
Revised: On CIFAR-10 (Krizhevsky, 2009), we notice that its relative performance compared to Deep Ensembles seems to increase as the original architecture gets bigger.

Original: For ResNet-18, the method matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.
Revised: For ResNet-18, the method matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.

Original: Using ResNet-50, both models seem to perform equivalently, while PackedEnsembles slightly outperforms Deep Ensembles in terms of classification performance with Wide
Revised: Using ResNet-50, both models seem to perform equivalently, while PackedEnsembles slightly outperforms Deep Ensembles in terms of classification performance with Wide

Original: ResNet28-10.
Revised: ResNet28-10.

Original: On CIFAR-100, Packed-Ensembles is outperformed by Deep Ensembles on ResNet-18.
Revised: On CIFAR-100, Packed-Ensembles is outperformed by Deep Ensembles on ResNet-18.

Original: Yet, we argue that ResNet-18 architecture does not have enough representation capacity to be divided into  subnetworks for CIFAR-100.
Revised: Yet, we argue that ResNet-18 architecture does not have enough representation capacity to be divided into [Table] [Table] subnetworks for CIFAR-100.

Original: Indeed, when we look at the results of ResNet-50, we can see that
Revised: Indeed, when we look at the results of ResNet-50, we can see that

Original: [Table]

Original: [Table]

Original: Packed-Ensembles has better results than Deep Ensembles.
Revised: Packed-Ensembles has better results than Deep Ensembles.

Original: This demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.
Revised: This demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.

Original: In appendix F, we discuss the influence of the representation capacity.
Revised: In appendix D, we discuss the influence of the representation capacity.

Original: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.
Revised: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.

Original: 4.1. S TUDY ON THE PARAMETERS α AND γ
Revised: 4.1. S TUDY ON THE PARAMETERS α AND γ

Original: Table 1 reports results only for α = 2 and γ = 2 , however depending on the task, the architecture used and the available memory, one might want to tune those parameters to fit more precisely its needs.
Revised: Table 1 reports results only for α = 2 and γ = 2 , however depending on the task, the architecture used and the available memory, one might want to tune those parameters to fit more precisely its needs.

Original: Figures 6 and 7 showcase the performance evolution of the Packed-Ensembles as the α values varies.
Revised: Figures 6 and 7 showcase the performance evolution of the Packed-Ensembles as the α values varies.

Original:  D ISCUSSIONS
Revised:  D ISCUSSIONS

Original: We have shown that Packed-Ensembles has interesting properties, especially by providing a similar quality of Uncertainty Quantification as Deep Ensemble while using a reduced architecture and computing cost.
Revised: We have shown that Packed-Ensembles has interesting properties, especially by providing a similar quality of Uncertainty Quantification as Deep Ensemble while using a reduced architecture and computing cost.

Original: Several questions can be raised and we conducted some studies, detailed in the
Revised: Several questions can be raised and we conducted some studies, detailed in the

Original: Appendix section, in order to bring possible answers.
Revised: Appendix section, in order to bring possible answers.

Original: [Table]
Revised: [Table]

Original: Discussion on the sparsity As described in section 3, one could interpret Packed-Ensembles as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.
Revised: Discussion on the sparsity As described in section 3, one could interpret Packed-Ensembles as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.

Original: In appendix C, by using a simplified model, we are able to bound the approximation error based on the Kullback-Leibler divergence between the Deep Ensemble and its pruned version.
Revised: In Appendix C, by using a simplified model, we are able to bound the approximation error based on the Kullback-Leibler divergence between the Deep Ensemble and its pruned version.

Original: This bound depends on the density of ones in the mask p , and more specifically, depends on the terms ( p − 1) , p ( p − 1) and ( p − 1) 2 /p 2 .
Revised: This bound depends on the density of ones in the mask p , and more specifically, depends on the terms ( p − 1) , p ( p − 1) and ( p − 1) 2 /p 2 .

Original: By manipulating these terms, corresponding to modifying the number of estimators M , the number of groups γ and the dilatation faction α , we could theoretically be able to control the approximation error.
Revised: By manipulating these terms, corresponding to modifying the number of subnetworks M , the number of groups γ and the dilatation faction α , we could theoretically be able to control the approximation error.

Original: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two main sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.
Revised: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two main sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.

Original: A last source of stochasticity is introduced during the training phase by the non-deterministic behaviour of the backward algorithms.
Revised: A last source of stochasticity is introduced during the training phase by the non-deterministic behaviour of the backward algorithms.

Original: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.
Revised: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.

Original: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seem to emerge to predict the best combination.
Revised: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seem to emerge to predict the best combination.

Original: Especially, we highlight that even the only use of nondeterministic algorithms introduces enough diversity between each estimator of the ensemble.
Revised: Especially, we highlight that even the only use of nondeterministic algorithms introduces enough diversity between each subnetwork of the ensemble.

Original: Ablation study We perform ablations studies in order to assess the impact of the parameters M , γ , and α on the performances of Packed-Ensembles.
Revised: Ablation study We perform ablations studies in order to assess the impact of the parameters M , γ , and α on the performances of Packed-Ensembles.

Original: The details of this study are given in Appendix
Revised: The details of this study are given in Appendix

Original: D. No explicit behavior appears from the results we obtained.
Revised: D. No explicit behavior appears from the results we obtained.

Original: A trend seems to show that a higher number of subnetworks is helpful in obtaining a better OOD detection, but the improvement in terms of AUPR is not significant.
Revised: A trend seems to show that a higher number of subnetworks is helpful in obtaining a better OOD detection, but the improvement in terms of AUPR is not significant.

Original: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.
Revised: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.

Original: This translates into an expected lower number of operations.
Revised: This translates into an expected lower number of operations.

Original: However, a study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.
Revised: However, a study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.

Original: Yet, this also hints that the actual speedup is not optimal despite the important acceleration offered by 16 bits floating points.
Revised: Yet, this also hints that the actual speedup is not optimal despite the important acceleration offered by 16 bits floating points.

Original: OOD criteria The criterion for discriminating OOD elements is based on the maximum softmax probability.
Revised: OOD criteria The criterion for discriminating OOD elements is based on the maximum softmax probability.

Original: However, this criterion is not unique, and others can be used, such as the Mutual Information, the maximum logit, or the Shannon entropy of the mean prediction.
Revised: However, this criterion is not unique, and others can be used, such as the Mutual Information, the maximum logit, or the Shannon entropy of the mean prediction.

Original: Although no relationship is expected between this criterion and the method of Packed-Ensembles, we obtained different performances in OOD detection according to the selected criterion.
Revised: Although no relationship is expected between this criterion and the method of Packed-Ensembles, we obtained different performances in OOD detection according to the selected criterion.

Original: The results are detailed in appendix E, and they show that an approach based on the maximum logit seems to give the best results in detecting OOD.
Revised: The results are detailed in Appendix E, and they show that an approach based on the maximum logit seems to give the best results in detecting OOD.

Original: It should be noted that this discussion focuses primarily on CIFAR-100 and that the notion of OOD depends on the training distribution.
Revised: It should be noted that this discussion focuses primarily on CIFAR-100 and that the notion of OOD depends on the training distribution.

Original: Such a discussion does not necessarily gen- eralize to all datasets.
Revised: Such a discussion does not necessarily generalize to all datasets.

Original: Indeed, preliminary results have shown that Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.
Revised: Indeed, preliminary results have shown that Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Ensembles and uncertainty quantification.
Revised: Ensembles and uncertainty quantification.

Original: Bayesian Neural Networks (BNNs) (MacKay, 1992;
Revised: Bayesian Neural Networks (BNNs) (MacKay, 1992;

Original: Neal, 1995) are the cornerstone and main source of inspiration for uncertainty quantification in deep learning.
Revised: Neal, 1995) are the cornerstone and main source of inspiration for uncertainty quantification in deep learning.

Original: In spite of the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain difficult to scale and train for large DNN architectures (Dusenberry et al., 2020).
Revised: In spite of the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain difficult to scale and train for large DNN architectures (Dusenberry et al., 2020).

Original: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of
Revised: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of

Original: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).
Revised: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).

Original: DE are currently the best performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: DE are currently the best performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: Efficient ensembles.
Revised: Efficient ensembles.

Original: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost have inspired a large cohort of approaches aiming to mitigate it.
Revised: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost have inspired a large cohort of approaches aiming to mitigate it.

Original: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork specific parameters trained in parallel.
Revised: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork specific parameters trained in parallel.

Original: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.
Revised: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.

Original: A single network can be used in ensemble-mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).
Revised: A single network can be used in ensemble-mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).

Original: Liu et al.
Revised: Liu et al.

Original: (2022) leverage the sparse networks training algorithm of Mocanu et al.
Revised: (2022) leverage the sparse networks training algorithm of Mocanu et al.

Original: (2018) to produce ensembles of sparse networks.
Revised: (2018) to produce ensembles of sparse networks.

Original: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020), by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).
Revised: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020), by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).

Original: However most of these approaches require multiple forward passes.
Revised: However most of these approaches require multiple forward passes.

Original: Neural network compression.
Revised: Neural network compression.

Original: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by-design, relying on e.g., channel shuffling (Zhang &
Revised: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by-design, relying on e.g., channel shuffling (Zhang &

Original: Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.
Revised: Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.

Original: Some of the most popular architectures the leverage such models are: SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b), MobileNet-v3 (Howard et al., 2019).
Revised: Some of the most popular architectures the leverage such models are: SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b), MobileNet-v3 (Howard et al., 2019).

Original: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;
Revised: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;

Original: Louizos et al., 2018; Frankle & Carbin, 2018;
Revised: Louizos et al., 2018; Frankle & Carbin, 2018;

Original: Tartaglione et al., 2022).
Revised: Tartaglione et al., 2022).

Original: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency.
Revised: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency.

Original: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.
Revised: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.

Original: Grouped convolutions.
Revised: Grouped convolutions.

Original: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.
Revised: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.

Original: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.
Revised: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.

Original: Xie et al.
Revised: Xie et al.

Original: (2017) demonstrate that using grouped convolutions lead to accuracy improvements and model complexity reduction.
Revised: (2017) demonstrate that using grouped convolutions lead to accuracy improvements and model complexity reduction.

Original: So far, grouped convolutions have been used primarily for computational efficiency.
Revised: So far, grouped convolutions have been used primarily for computational efficiency but also to compute multiple output branches in parallel (Chen & Shrivastava, 2020).

Original: Here, we re-purpose them for delineating multiple subnetworks within a network and for efficiently training an ensemble of such subnetworks.
Revised: Here, we re-purpose them for delineating multiple subnetworks within a network and for efficiently training an ensemble of such subnetworks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a new ensemble framework: Packed-Ensembles, which is able to approximate Deep
Revised: We propose a new ensemble framework: Packed-Ensembles, which is able to approximate Deep

Original: Ensembles in terms of uncertainty quantification and accuracy.
Revised: Ensembles in terms of uncertainty quantification and accuracy.

Original: Our work provides several new findings: first, we show that ensembling independent small neural networks can be equivalent to ensembling independent deep neural networks.
Revised: Our work provides several new findings: first, we show that ensembling independent small neural networks can be equivalent to ensembling independent deep neural networks.

Original: Secondly, we demonstrate that not all sources of diversity are needed to improve the diversity of the ensemble.
Revised: Secondly, we demonstrate that not all sources of diversity are needed to improve the diversity of the ensemble.

Original: Thirdly, we show that Packed-Ensembles is more stable than a single DNN.
Revised: Thirdly, we show that Packed-Ensembles is more stable than a single DNN.

Original: Fourthly, we highlight that there is a trade-off between the accuracy and the size of the parameters, and Packed-Ensembles allows us to have flexible and efficient ensembling.
Revised: Fourthly, we highlight that there is a trade-off between the accuracy and the size of the parameters, and Packed-Ensembles allows us to have flexible and efficient ensembling.

Original: In the future, we intend to explore Packed-Ensembles for more complex tasks.
Revised: In the future, we intend to explore Packed-Ensembles for more complex tasks.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: Alongside this paper, we provide the source code of Packed-Ensembles layers.
Revised: Alongside this paper, we provide the source code of Packed-Ensembles layers.

Original: Moreover, two notebooks enable the training of Packed-Ensembles based on the ResNet-50 architecture, both on
Revised: Moreover, two notebooks enable the training of Packed-Ensembles based on the ResNet-50 architecture, both on

Original: CIFAR-10 and CIFAR-100 (public datasets).
Revised: CIFAR-10 and CIFAR-100 (public datasets).

Original: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.
Revised: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.

Original: Furthermore, it should be noted that the source code contains two PyTorch Module classes to efficiently produce Packed-Ensembles.
Revised: Furthermore, it should be noted that the source code contains two PyTorch Module classes to efficiently produce Packed-Ensembles.

Original: The idea would indeed be to release a Python package to provide easier access to Packed-Ensembles layers.
Revised: The idea would indeed be to release a Python package to provide easier access to Packed-Ensembles layers.

Original: A readme file at the root of the project details how to install and run experiments.
Revised: A readme file at the root of the project details how to install and run experiments.

Original: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).
Revised: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).

Original:  E THICS
Revised:  E THICS

Original: The purpose of this paper is to provide a method for a better estimation of Deep Learning model uncertainty.
Revised: The purpose of this paper is to provide a method for a better estimation of Deep Learning model uncertainty.

Original: Nevertheless, we acknowledge their limitation, which could become critical when applied to safety systems for instance.
Revised: Nevertheless, we acknowledge their limitation, which could become critical when applied to safety systems for instance.

Original: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.
Revised: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.

Original: We show the limitations of our approach in several experiments.
Revised: We show the limitations of our approach in several experiments.

Original: Many more validation and verification steps would be need before real-world deployment to ensure robustness to various unknown situations, corner-cases, adversarial attacks, biases.
Revised: Many more validation and verification steps would be need before real-world deployment to ensure robustness to various unknown situations, corner-cases, adversarial attacks, biases.

Original: T ABLE OF C ONTENTS 
Revised: T ABLE OF C ONTENTS - S UPPLEMENTARY M ATERIALS

Original: A Notations
Revised: A Notations

Original: B Implementation details
Revised: B Implementation details

Original: B.1 General Considerations . .

Original: B.2 Masksembles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C Discussion on the Sparsity
Revised: C Discussion on the sparsity

Original: D Ablation Study
Revised: D Ablation Study

Original: E Discussion about OOD criteria
Revised: E Discussion about OOD criteria

Original: F Discussion about the sources of stochasticity
Revised: F Discussion about the sources of stochasticity

Original: G Discussion about the subnetworks
Revised: G Discussion about the subnetworks

Original: H Discussion about the training velocity
Revised: H Discussion about the training velocity

Original: I Distribution shift
Revised: I Distribution shift

Original:  Stabilization of the performance
Revised: J Stabilization of the performance

Original: K On the Equivalence between sequential training and Packed-Ensembles
Revised: K On the equivalence between sequential training and Packed-Ensembles

Original: L Using groups is not sufficient to provide equivalent results to Packed-Ensembles
Revised: L Using groups is not sufficient to provide equivalent results to Packed-Ensembles

Original: M Efficiency of the networks trained on ImageNet
Revised: M Efficiency of the networks trained on ImageNet

Original: N Regression
Revised: N Regression

Original: A N OTATIONS
Revised: A N OTATIONS

Original: We summarize the main notations used in the paper in Table3.
Revised: We summarize the main notations used in the paper in Table

Original: Our code is implemented in PyTorch (Paszke et al., 2019) using the PyTorch Lightning framework.
Revised: [Table] [Table]

Original: The code will be made publicly available after the anonymity period. To ensure that the layers convey sufficient information and are not weakened by groups, we have chosen to set a constant minimum number of channels per group to 64 for all experiments presented in the paper. Experiments in which this minimum could play an important part and lead to confusions (see for instance PE-( 1 , 4 , 4 ) in Table 5) have not been conducted.
Revised: D

Original: [Table]
Revised: [Table]

Original: D = { ( x i , y i ) } |D| i
Revised: data samples

Original: [Table]
Revised: [Table]

Original: The set of |D| data samples and the corresponding labelsj , m ,
Revised: the corresponding labels

Original: [Table]
Revised: [Table]

Original: The index of the current layer, the current subnetwork, and the number of layersz
Revised: index of the current layer, the current subnetwork, and the number of layers

Original: [Table]
Revised: [Table]

Original: The preactivation feature map and output of the layer ( j − 1) /input of layer j
Revised: activation

Original: [Table]
Revised: [Table]

Original: The activation function (considered constant throughout the network)
Revised: constant throughout the network)

Original: [Table]
Revised: [Table]

Original: The featuremap and output of layer j , h j = ϕ ( z j )
Revised: feature

Original: [Table]
Revised: [Table]

Original: The height and width of the feature maps and output of layer j −
Revised: number of parameters

Original: [Table]
Revised: [Table]

Original: The number of channels of the feature maps and output of layerj −
Revised: layer

Original: [Table]
Revised: [Table]

Original: The number of parameters of layer j
Revised: batch size of the training procedure

Original: [Table]
Revised: [Table]

Original: The mask corresponding to the layer j of the subnetwork m
Revised: The mask corresponding to the layer j of the subnetwork m

Original: [Table]
Revised: [Table]

Original: The integer part function

Original: [Table]

Original: The 2D cross-correlation, the convolution and the Hadamard product
Revised: The 2D cross-correlation, the convolution and the Hadamard product

Original: [Table]
Revised: [Table]

Original: The size of the kernel of the layer j
Revised: The size of the kernel of the layer j

Original: [Table]
Revised: [Table]

Original: The number of subnetworks in an ensemble
Revised: The number of subnetworks in an ensemble

Original: [Table]
Revised: [Table]

Original: The prediction of the subnetwork number m concerning the input x
Revised: The prediction of the subnetwork number m concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The prediction of the ensemble concerning the input x
Revised: The prediction of the ensemble concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The width-augmentation factor of Packed-Ensembles
Revised: The width-augmentation factor of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The number of subgroups of Packed-Ensembles
Revised: The number of subgroups of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The set of weights of the subnetwork m with a width factor α
Revised: The set of weights of the subnetwork m with a width factor α

Original: [Table]
Revised: [Table]

Original: The weights of layer j with γ groups and a width factor α
Revised: The weights of layer j with γ groups and a width factor α

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: General Considerations. Our code is implemented in PyTorch (Paszke et al., 2019) using the

Original: [Table] [Table] [Table] [Table]
Revised: PyTorch Lightning framework.

Original: Medium
Revised: The code will be made publicly available after the anonymity period.

Original: Table 4 summarizes all the hyperparameters used in the paper for CIFAR-10 and CIFAR-100.
Revised: Table 4 summarizes all the hyperparameters used in the paper for CIFAR-10 and CIFAR-100.

Original: HFlip denotes the classic horizontal flip.
Revised: In all cases, we use SGD combined with a multistep-learning-rate scheduler multiplying the rate by γ -lr at each milestone.

Original: The BatchEnsemble based on ResNet-50 uses a smaller learning rate of 0.08 instead of 0.1 for stability.
Revised: Note that BatchEnsemble based on ResNet-50 uses a smaller learning rate of 0.08 instead of 0.1 for stability.

Original: The medium data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) 
Revised: The medium data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0 .

Original:  using timm’s augmentation classes 3 , with alphas respectively 0.5 and 0.2.
Revised: 5 switch probability and using timm’s augmentation classes 3 , with coefficients respectively 0.5 and 0.2.

Original: We also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1 in this case.
Revised: We also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1 in this case.

Revised: To ensure that the layers convey sufficient information and are not weakened by groups, we have chosen to set a constant minimum number of channels per group to 64 for all experiments presented in the paper.

Revised: In this case, we choose the maximum number of groups respecting this constraint.

Revised: Experiments in which this minimum could play a significant part and lead to confusions are not presented (see for instance PE-( 1 , 4 , 4 ) in Table 5).

Original: For ImageNet, we use the A3 procedure from Wightman et al.
Revised: For ImageNet, we use the A3 procedure from Wightman et al.

Original: (2021) for all models.
Revised: (2021) for all models.

Revised: Training with the exact A3 procedure was not always possible.

Revised: Refer to the specific subsection for more details.

Original: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).
Revised: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).

Revised: We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).

Original: B. M ASKSEMBLES
Revised: Masksembles.

Original: Concerning Masksembles (Durasov et al., 2021), we use the code proposed by the authors 4 .
Revised: We use the code proposed by (Durasov et al., 2021) 4 .

Original: We modified the mask generation function using binary search as proposed by the authors, since it was unable to build masks for ResNet50x4.
Revised: We modified the mask generation function using binary search as proposed by the authors, since it was unable to build masks for ResNet50x4.

Original: We note that the code implies performing batch repeats at the start the forward passes.
Revised: We note that the code implies performing batch repeats at the start the forward passes.

Original: All the results regarding this technique are therefore computed with this specification. 
Revised: All the results regarding this technique are therefore computed with this specification. The

Original: The ResNet implementations are built using Masksemble2D layers with M=4 and a scale factor of 2 after each convolutions. 
Revised: ResNet implementations are built using Masksemble2D layers with M = 4 and a scale factor ofafter each convolutions. BatchEnsemble.

Revised: For BatchEnsemble, we use two different values for weight decay.

Revised: Table 4 provides the weight decay corresponding to the shared weights.

Revised: However, no weight decay is applied to the vectors S and R (which generate the rank-1 matrices).

Original:  C D ISCUSSION ON THE S PARSITY
Revised: ImageNet. The batch size of Masksembles ResNet-50x4 is reduced to 1120 because of memory constraints. Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence. C D ISCUSSION ON THE SPARSITY

Original: In this section, we are going to provide an estimation of the expected distance between a dense, fully-connected layer, and a sparse one.
Revised: In this section, we are going to provide an estimation of the expected distance between a dense, fully-connected layer, and a sparse one.

Original: For simplicity, we are here assuming to operate with a fully-connected layer.
Revised: For simplicity, we are here assuming to operate with a fully-connected layer.

Original: First, let us write our first proposition:
Revised: First, let us write our first proposition:

Original: Proposition C.1.
Revised: Proposition C.1.

Original: Given a fully connected layer j + 1 defined by:
Revised: Given a fully connected layer j + 1 defined by:

Original: [Equation]
Revised: [Equation]

Original: and its approximation defined by:
Revised: and its approximation defined by:

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:
Revised: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜
Revised: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜

Original: z
Revised: z

Original: j +1 ( c ) .
Revised: j +1 ( c ) .

Original: A plot for (8) is provided in Figure 5.
Revised: A plot for (8) is provided in Figure 5.

Original: Proof.
Revised: Proof.

Original: To prove Prop.
Revised: To prove Prop.

Original: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,
Revised: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,

Original: z j +1 will be as well gaussian-distributed.
Revised: z j +1 will be as well gaussian-distributed.

Original: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:
Revised: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:

Original: [Equation]
Revised: [Equation]

Original: and the variance is:
Revised: and the variance is:

Original: [Equation]
Revised: [Equation]

Original: If we assume Σ( i, k ) = 0 ∀ i ̸
Revised: If we assume Σ( i, k ) = 0 ∀ i ̸

Original: = k , (10) simplifies into:
Revised: = k , (10) simplifies into:

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the case with the mask, similarly as presented in (3):
Revised: Let us now consider the case with the mask, similarly as presented in (3):

Original: [Equation]
Revised: [Equation]

Original: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).
Revised: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).

Original: In the limit of large C j , we know that ˜
Revised: In the limit of large C j , we know that ˜

Original: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:[Equation] [Equation] Hence
Revised: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:

Original: [Equation]
Revised: [Equation]

Original:  we have:
Revised: Hence, we have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In order to assess the dissimilarity between z and ˜ z
Revised: In order to assess the dissimilarity between z and ˜ z

Original: [Equation]
Revised: [Equation]

Original: we can write the Kullback–Leibler divergence:
Revised: we can write the Kullback–Leibler divergence:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Straightforwardly we can write the inequality:
Revised: Straightforwardly we can write the inequality:

Original: [Equation]
Revised: [Equation]

Original: According to (16) we can write:
Revised: According to (16) we can write:

Original: [Equation]
Revised: [Equation]

Original:  

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Table]

Original: Since we know that
Revised: Since we know that

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure] [Equation] [Figure] [Table]
Revised: we can also write:

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Figure] [Table] [Figure]
Revised: Finally

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] we can also write:
Revised: according to: (15)

Original: [Equation]
Revised: [Equation]

Original: Finally, according to: (15)
Revised: finding back (8)

Original: [Equation]
Revised: [Equation]

Original: finding back (8).
Revised: [Figure] [Figure]

Original: D A BLATION S TUDY
Revised: D A BLATION S TUDY

Original: Our algorithm mainly depends on three hyperparameters.
Revised: Our algorithm mainly depends on three hyperparameters.

Original: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.
Revised: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.

Original: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.
Revised: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.

Original: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performances, in terms of accuracy and AUPR.
Revised: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performances, in terms of accuracy and AUPR.

Original: We also note that the results are stable with γ .
Revised: We also note that the results are stable with γ .

Original: Moreover, we notice that the resulting accuracy tends to increase with α until it reaches a plateau.
Revised: Moreover, we notice that the resulting accuracy tends to increase with α until it reaches a plateau.

Original: These statement are confirmed by the results in Table 5.
Revised: These statement are confirmed by the results in Table 5.

Original: E D ISCUSSION ABOUT OOD CRITERIA
Revised: E D ISCUSSION ABOUT OOD CRITERIA

Original: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.
Revised: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.

Original: Similarly to Bayesian Neural
Revised: Similarly to Bayesian Neural

Original: [Figure]

Original: [Table]

Original: Network, one can take the softmax outputs of posterior predictive distribution, which define the
Revised: Network, one can take the softmax outputs of posterior predictive distribution, which define the

Original: MSP = max y i
Revised: MSP = max y i

Original: { P ( y i | x , D ) } .
Revised: { P ( y i | x , D ) } .

Original: The MSP can also be used for classical DNN, yet in this case, we use the conditional likelihood instead of the posterior distribution.
Revised: The MSP can also be used for classical DNN, yet in this case, we use the conditional likelihood instead of the posterior distribution.

Original: One can also use the Maximum Logit (ML) as an uncertainty criterion and the entropy of the posterior predictive distribution as an uncertainty criterion, which is defined by Ent.
Revised: One can also use the Maximum Logit (ML) as an uncertainty criterion and the entropy of the posterior predictive distribution as an uncertainty criterion, which is defined by Ent.

Original: =
Revised: =

Original: H ( P ( y i | x , D )) with H being the entropy function.
Revised: H ( P ( y i | x , D )) with H being the entropy function.

Original: Another metric is the mutual information(MI) between two random variables, which is defined by: MI = H ( P ( y i | x , D )) − 
Revised: Another metric is the mutual information between two random variables, which is defined by: MI = H ( P ( y i | x , D )) − 1 M M − 1 (cid:80) m =0 H ( P ( y | θ α,m , x )) . It represents a

Original: [Equation] [Equation]
Revised: [Figure] [Table] [Table]

Original: , x )) . It represents a measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.
Revised: measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.

Original: The last metric used in active learning is the variation ratio (Beluch et al., 2018) which measures the dispersion of a nominal variable, and is calculated as the proportion of predicted class labels that are not the modal class prediction.
Revised: The last metric used in active learning is the variation ratio (Beluch et al., 2018) which measures the dispersion of a nominal variable [Table] and is calculated as the proportion of predicted class labels that are not the modal class prediction.

Original: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.
Revised: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.

Original: In Table 6, the results for the different metrics are reported.
Revised: In Table 6, the results for the different metrics are reported.

Original: We note that ML seems to be the best metric to detect OOD.
Revised: We note that ML seems to be the best metric to detect OOD.

Original: This metric is followed by Ent.
Revised: This metric is followed by Ent.

Original: and then MI .
Revised: and then MI .

Original: Note that v , which is widely used in active learning, does not seem to be effective in detecting OOD.
Revised: Note that v , which is widely used in active learning, does not seem to be effective in detecting OOD.

Original: This shows us that it is essential to use a good criterion in addition to good ensembling.
Revised: This shows us that it is essential to use a good criterion in addition to good ensembling.

Original: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY
Revised: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY

Original: As written in the introduction of the paper, diversity is essential to the success of ensembling.
Revised: As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection.

Original: Three main sources can induce weight diversity and therefore diversity in the function space.
Revised: Three main sources can induce weight diversity and therefore diversity in the function space during the training.

Original: These sources are the initialization of the weights, the composition of the batches and the use of non-deterministic backpropagation algorithms 5 .
Revised: These sources are the initialization of the weights, the composition of the batches and the use of non-deterministic backpropagation algorithms 5 .

Original: On Table 7, we measure themean performance over 5 experiments
Revised: On Table 7, we measure the performance and diversity of Packed-Ensembles trained on CIFAR-100.

Original: [Table]
Revised: This diversity is measured by the mutual information and is twofold: we compute the in-distribution mutual information (ID MI ) on the test set of CIFAR-100 and the OOD mutual information (OOD MI ) on SVHN.

Original:  in accuracy, NLL, calibration and OOD-Detection for all major configurations on CIFAR 100 for
Revised: of the mean performance of Packed-Ensembles overexperiments in accuracy, NLL, calibration, and OOD detection on CIFAR-100.

Original: Packed-Ensembles.
Revised: Concerning the performance, we compute the accuracy, ECE, and AUPR which are proxies of the quality of this diversity. Results of Table 7 lead to several takeaways. First, they hint that there is no clear best set of trivial sources of stochasticity. Except for the first (and greyed) line, which corresponds to ensembling completely identical networks (the training being totally deterministic which is confirmed by the null MI), the results seem equivalent in diversity (via the mutual information) and in ID/OOD performance.

Original: Results on Table 7 show that the use of non-deterministic algorithms can be sufficient to generate diversity.
Revised: Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity.

Original: It was noted that this effect does not always appear depending on the selected architecture and the precision used ( float16 ,  float32 ).
Revised: It was noted that this effect does not always happen depending on the selected architecture and the precision used ( float16 , or float32 ).

Original: In practice, we  use nondeterministic backpropagation algorithms as well as different inializations to ensure enough stochasticity but also for programming convenience and training speed.
Revised: Given that there is no emerging best set of stochasticity, we choose in practice to use the faster non-deterministic backpropagation algorithms as well as different initializations to ensure enough stochasticity and for programming convenience.

Original: The accuracy, ECE and AUPR are given as a proxy of the quality of the diversity while the mutual information provides a quantification of this diversity. We confirm that using deterministic backpropagation algorithm, the same initialization and the same batches lead to the same weights and therefore the same values in the function space.
Revised: [Table]

Original: G D ISCUSSION ABOUT THE SUBNETWORKS
Revised: G D ISCUSSION ABOUT THE SUBNETWORKS

Original: The discussion between the width and depth of deep neural networks is an essential topic of research.
Revised: The discussion between the width and depth of deep neural networks is an essential topic of research.

Original: Researchers have focused on the correct approaches to increase the depth of DNN, which in turn increases the accuracy.
Revised: Researchers have focused on the correct approaches to increase the depth of DNN [Table] which in turn increases the accuracy.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2020) demonstrate that the width and depth are connected to the model capacity and enable the DNN to learn block structures which lead to good accuracy.
Revised: (2020) demonstrate that the width and depth are connected to the model capacity and enable the DNN to learn block structures which lead to good accuracy.

Original: Hence, by splitting the width, chances are that we could decrease the model capacity.
Revised: Hence, by splitting the width, chances are that we could decrease the model capacity.

Original: Deep neural networks are models with huge amounts of parameters, works like the lottery ticket hypothesis (Frankle & Carbin, 2018) suggest that these DNNs are heavily overparameterized.
Revised: Deep neural networks are models with huge amounts of parameters, works like the lottery ticket hypothesis (Frankle & Carbin, 2018) suggest that these DNNs are heavily overparameterized.

Original: Thus it is possible to kill up to 80% of the neurons without losing too much performance.
Revised: Thus it is possible to kill up to 80% of the neurons without losing too much performance.

Original: MIMO builds on this assumption to ensure that each network drives several networks simultaneously.
Revised: MIMO builds on this assumption to ensure that each network drives several networks simultaneously.

Original: However, unlike in our work, the boundaries between the networks are not clearly defined.
Revised: However, unlike in our work, the boundaries between the networks are not clearly defined.

Original: Thus, the same neuron can be used for multiple different subnetworks (DNNs) in the same ensemble.
Revised: Thus, the same neuron can be used for multiple different subnetworks (DNNs) in the same ensemble.

Original: In our case, we clearly assign each neuron to a DNN from the ensemble.
Revised: In our case, we clearly assign each neuron to a DNN from the ensemble.

Original: This way, the DNNs do not get mixed up and can learn an independent representation.
Revised: This way, the DNNs do not get mixed up and can learn an independent representation.

Original: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.
Revised: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.

Original: This decomposition may seem crude.
Revised: This decomposition may seem crude.

Original: However, it allows us to better parallelize Packed-Ensembles for training and inference.
Revised: However, it allows us to better parallelize Packed-Ensembles for training and inference.

Original: To overcome the fact that our networks are not wide enough if M is too large, we have added an alpha hyperparameter that can increase the width of the subnetworks.
Revised: To overcome the fact that our networks are not wide enough if M is too large, we have added an alpha hyperparameter that can increase the width of the subnetworks.

Original: In Figure 8, we study the influence of the width of the subnetworks.
Revised: In Figure 8, we study the influence of the width of the subnetworks.

Original: First of all, we notice that the accuracy increases with the width, while the AUPR seems relatively constant.
Revised: First of all, we notice that the accuracy increases with the width, while the AUPR looks relatively constant.

Original: This seems to confirm the importance of the alpha parameter in balancing the width of the DNN.
Revised: This seems to confirm the importance of the alpha parameter in balancing the width of the DNN.

Original: Furthermore, reducing the width does not appear to reduce the accuracy significantly.
Revised: Furthermore, reducing the width does not appear to reduce the accuracy significantly.

Original: Thus, this justifies our choice to split the width of the DNN to produce several subnetworks since the uncertainty quantification will be constant, and the accuracy will not drop drastically. 
Revised: Thus, this justifies our choice to split the width of the DNN to produce several subnetworks since the uncertainty quantification will be constant, and the accuracy will not drop drastically. Moreover,

Original: Moreover, the addition of the alpha parameter allows us to add a new degree of freedom to our ensemble and enables it to increase its accuracy. [Figure]
Revised: [Figure] [Table] the addition of the alpha parameter allows us to add a new degree of freedom to our ensemble and enables it to increase its accuracy

Original: [Table]
Revised: [Table]

Original: H D ISCUSSION ABOUT THE TRAINING VELOCITY
Revised: H D ISCUSSION ABOUT THE TRAINING VELOCITY

Original: Our experiments show that grouped convolutions are not as fast as they could theoretically be, and confirm the statements made by many PyTorch and TensorFlow users 6 .
Revised: Our experiments show that grouped convolutions are not as fast as they could theoretically be [Table] and confirm the statements made by many PyTorch and TensorFlow users 6 .

Original: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed
Revised: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed

Original: Precision (AMP) and cuDNN benchmark flags when training an Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.
Revised: Precision (AMP) and cuDNN benchmark flags when training an Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.

Original: AMP will also divide the VRAM usage by two while yielding equally good results.
Revised: AMP will also divide the VRAM usage by two while yielding equally good results.

Original: Future improvements of PyTorch grouped convolutions should help this method develop its full potential, increasing its current assets.
Revised: Future improvements of PyTorch grouped convolutions should help Packed-Ensembles develop its full potential, increasing its current assets.

Original: We note in Table 8 that using float16 , Packed-Ensembles is only 1 .
Revised: We note in Tablethat using float16 , Packed-Ensembles is only 1 .

Original: 6 × slower than the single model during inference.
Revised: 6 × slower than the single model during inference.

Original: Furthermore, Packed-Ensembles is only 2 .
Revised: Furthermore, Packed-Ensembles is only 2 .

Original: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.
Revised: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.

Original: I D ISTRIBUTION SHIFT
Revised: I D ISTRIBUTION SHIFT

Original: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.
Revised: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.

Original: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are 5 levels of perturbation called ”severity”, from 1, the weakest to 5, the strongest.
Revised: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are 5 levels of perturbation called ”severity”, from 1, the weakest to 5, the strongest.

Original: In real world scenario, distributional shift is crucial as explained by (Ovadia et al., 2019), and it is crucial to study how much a model prediction shifts from the original training data distribution.
Revised: In real world scenario, distributional shift is crucial as explained by (Ovadia et al., 2019), and it is crucial to study how much a model prediction shifts from the original training data distribution.

Original: Thanks to Figure 9 we notice that
Revised: Thanks to Figure 9 we notice that

Original:  Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.[Figure] [Table]
Revised: [Figure] Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.

Original: J S TABILIZATION OF THE PERFORMANCE
Revised: J S TABILIZATION OF THE PERFORMANCE

Original: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute a variance.
Revised: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute a variance.

Original: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .
Revised: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .

Original: The use of ensemble strategies shrinks the standard variation to 0 .
Revised: The use of ensemble strategies shrinks the standard variation to 0 .

Original: 43% for Deep Ensembles and 0 .
Revised: 43% for Deep Ensembles and 0 .

Original: 19% for Packed-Ensembles.
Revised: 19% for Packed-Ensembles.

Original: Thus it seems that Packed-Ensembles make DNN predictions more stable in addition to improving accuracy and uncertainty quantification.
Revised: Thus it seems that Packed-Ensembles makes DNN predictions more stable in addition to improving accuracy and uncertainty quantification.

Original: This result is interesting as it seems to contradict (Neal et al., 2018) who claim that wider DNNs have a smaller variance.
Revised: This result is interesting as it seems to contradict Neal et al. (2018) who claim that wider DNNs have a smaller variance.

Original: This stability might come from the ensembling.
Revised: This stability might come from the ensembling.

Original: K O N THE E QUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES
Revised: K O N THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES

Original: The sequential training of Deep Ensembles differs significantly from the training procedure of
Revised: The sequential training of Deep Ensembles differs significantly from the training procedure of

Original: Packed-Ensembles.
Revised: Packed-Ensembles.

Original: The main differences lie in the subnetworks’ batch composition and the best models’ selection.
Revised: The main differences lie in the subnetworks’ batch composition and the best models’ selection.

Original: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.
Revised: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.

Original: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see
Revised: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see

Original: Appendix F for more details).
Revised: Appendix F for more details).

Original: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.
Revised: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.

Original: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.[Table]
Revised: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.

Original: L U SING GROUPS IS NOT SUFFICIENT TO PROVIDE EQUIVALENT RESULTS TO P ACKED -E NSEMBLES
Revised: L U SING GROUPS IS NOT SUFFICIENT TO PROVIDE EQUIVALENT RESULTS TO P ACKED -E NSEMBLES

Original: To make sure that our results cannot benot simply explained by the use of groups [Table] we compare on CIFAR-10 our Packed-Ensembles to a single ResNeXt-50 (32 × 4d) proposed by Xie et al.
Revised: To make sure that our results cannot be simply explained by the use of groups, we compare PackedEnsembles to a single ResNeXt-50 (32 × 4d) (Xie et al., 2017) in Table 9.

Original: (2017) in

Original: Table 9.

Original: The ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of each block,  which are therefore not independent.
Revised: The ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of each block, [Table] [Table] which are therefore not independent.

Original: The training optimization procedures are the same as well as the data-augmentation strategies and are all detailed in Appendix B.
Revised: The training optimization procedures are the same as well as the data-augmentation strategies and are all detailed in Appendix B.

Original: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET
Revised: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET

Original: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.
Revised: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.

Revised: PE-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.

Original: N R EGRESSION
Revised: N R EGRESSION

Original: To generalize our work, we propose to study regression tasks.
Revised: To generalize our work, we propose to study regression tasks.

Original: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.
Revised: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.

Original: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (21) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).
Revised: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (21) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).

Original: One output corresponds to the mean of the predicted gaussian distribution, and the softmax applied on the secondone is its variance.
Revised: One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance.

Original: The ensemble mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ 
Revised: The ensemble mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ σ θ ( x

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ) 2 =

Original: [Equation]
Revised: [Equation]

Original: Lakshmi
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: narayanan et al., 2017).
Revised: ) (Lakshminarayanan et al., 2017).

Original: [Equation]
Revised: [Equation]

Original: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.
Revised: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.

Original: The subnetworks of these methods are based ona multi-layer perceptron with a single hidden layer, containing 400 neurons for the bigger Protein dataset and 200 for the others, and a ReLU nonlinearity.
Revised: The subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, containing 400 neurons for the bigger Protein dataset and 200 for the others, and a ReLU non-linearity.

Original: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.
Revised: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.

Original: [Table]
Revised: [Table]