Original: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION
Revised: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation and out-of-distribution detection.
Revised: Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation and out-of-distribution detection.

Original: However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties.
Revised: However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties.

Original: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.
Revised: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.

Original: We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds.
Revised: We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds.

Original: PE is designed to operate within the memory limits of a standard neural network.
Revised: PE is designed to operate within the memory limits of a standard neural network.

Original: Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity and performs equally well in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.
Revised: Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity and performs equally well in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.

Original: We make our code available at github.com/ENSTAU2IS/torch-uncertainty
Revised: We make our code available at github.com/ENSTAU2IS/torch-uncertainty

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose exceptionally high reliability and performance requirements across a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.
Revised: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose exceptionally high reliability and performance requirements across a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.

Original: Despite significant improvements in performance in recent years, vanilla Deep Neural Networks (DNNs) still exhibit several shortcomings, notably overconfidence in both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).
Revised: Despite significant improvements in performance in recent years, vanilla Deep Neural Networks (DNNs) still exhibit several shortcomings, notably overconfidence in both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).

Original: Deep Ensembles (Lakshminarayanan et al., 2017) have emerged as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.
Revised: Deep Ensembles (Lakshminarayanan et al., 2017) have emerged as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.

Original: By averaging predictions or by voting, DE achieves high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.
Revised: By averaging predictions or by voting, DE achieves high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.

Original: Thanks to the simplicity and effectiveness of the ensembling strategy (Dietterich, 2000), DE have become widely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: Thanks to the simplicity and effectiveness of the ensembling strategy (Dietterich, 2000), DE have become widely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: DE meet most of the real-world application requirements except computational efficiency.
Revised: DE meet most of the real-world application requirements except computational efficiency.

Original: Specifically, DE are computationally demanding in terms of memory storage, number of operations, and inference time during both training and testing, as their costs grow linearly with the number of individuals.
Revised: Specifically, DE are computationally demanding in terms of memory storage, number of operations, and inference time during both training and testing, as their costs grow linearly with the number of individuals.

Original: Their computational costs are, therefore, prohibitive under tight hardware constraints.
Revised: Their computational costs are, therefore, prohibitive under tight hardware constraints.

Original: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020), ensemble-imitating layers (Wen et al., 2019;
Revised: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020), ensemble-imitating layers (Wen et al., 2019;

Original: Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016;
Revised: Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016;

Original: Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018), and efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).
Revised: Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018), and efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).

Original: These approaches typically improve storage usage, train cost, or inference time at the cost of lower accuracy and diversity in the predictions.
Revised: These approaches typically improve storage usage, train cost, or inference time at the cost of lower accuracy and diversity in the predictions.

Original: An essential property of ensembles to improve predictive uncertainty estimation is related to the diversity of its predictions.
Revised: An essential property of ensembles to improve predictive uncertainty estimation is related to the diversity of its predictions.

Original: Perrone & Cooper (1992) show that the independence of individuals is critical to the success of ensembling.
Revised: Perrone & Cooper (1992) show that the independence of individuals is critical to the success of ensembling.

Original: Fort et al.
Revised: Fort et al.

Original: (2019) argue that the diversity of DE, due to randomness from weight initialization, data augmentation and batching, and stochastic gradient updates, is superior to other efficient ensembling alternatives, despite their predictive performance boosts.
Revised: (2019) argue that the diversity of DE, due to randomness from weight initialization, data augmentation and batching, and stochastic gradient updates, is superior to other efficient ensembling alternatives, despite their predictive performance boosts.

Original: Few approaches manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes, and image throughput).
Revised: Few approaches manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes, and image throughput).

Original: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular, having a set of independent networks, in a computationally efficient manner.
Revised: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular, having a set of independent networks, in a computationally efficient manner.

Original: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.
Revised: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.

Original: We build upon this idea and devise a strategy based on small networks trying to match the performance of an ensemble of large networks.
Revised: We build upon this idea and devise a strategy based on small networks trying to match the performance of an ensemble of large networks.

Original: To this end, we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.
Revised: To this end, we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.

Original: The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models.
Revised: The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models.

Original: This method enables fast training and inference times while predictive uncertainty quantification is close to DE (Figure 1).
Revised: This method enables fast training and inference times while predictive uncertainty quantification is close to DE (Figure 1).

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured sparsity for Deep Ensembles; • We extensively evaluate PE regarding accuracy, calibration, OOD detection, and distribution shift on classification and regression tasks.
Revised: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured sparsity for Deep Ensembles; • We extensively evaluate PE regarding accuracy, calibration, OOD detection, and distribution shift on classification and regression tasks.

Original: We show that PE achieves state-of-the-art predictive uncertainty quantification.
Revised: We show that PE achieves state-of-the-art predictive uncertainty quantification.

Original: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.
Revised: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we present the formalism for this work and offer a brief background on grouped convolutions and ensembles of DNNs.
Revised: In this section, we present the formalism for this work and offer a brief background on grouped convolutions and ensembles of DNNs.

Original: Appendix A summarizes the main notations in Table 3.
Revised: Appendix A summarizes the main notations in Table 3.

Original: 2. B ACKGROUND ON CONVOLUTIONS
Revised: 2. B ACKGROUND ON CONVOLUTIONS

Original: The convolutional layer (LeCun et al., 1989) consists of a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C
Revised: The convolutional layer (LeCun et al., 1989) consists of a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C

Original: j +1 ×
Revised: j +1 ×

Original: C
Revised: C

Original: j × s 2 j
Revised: j × s 2 j

Original: [Figure]
Revised: [Figure]

Original: with C j , H j , W j three integers representing the number of channels, the height and the width of h j respectively.
Revised: with C j , H j , W j three integers representing the number of channels, the height and the width of h j respectively.

Original: C j +

Revised: C j +

Original: and s j are also two integers corresponding to the number of channels of h j
Revised: and s j are also two integers corresponding to the number of channels of h j

Original: + (the output of the layer) and the kernel size.
Revised: + (the output of the layer) and the kernel size.

Original: Finally, j is the layer’s index and will be fixed in the following formulae.
Revised: Finally, j is the layer’s index and will be fixed in the following formulae.

Original: The bias of convolution layers will be omitted in the following for simplicity.
Revised: The bias of convolution layers will be omitted in the following for simplicity.

Original: Hence the output value of the convolution layer, denoted ⊛ , is:
Revised: Hence the output value of the convolution layer, denoted ⊛ , is:

Original: [Equation]
Revised: [Equation]

Original: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ
Revised: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ

Original: ( z j ) with ϕ an activation function.
Revised: ( z j ) with ϕ an activation function.

Original: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in
Revised: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in

Original: ResNeXt (Xie et al., 2017) to train several DNN branches in parallel.
Revised: ResNeXt (Xie et al., 2017) to train several DNN branches in parallel.

Original: The grouped convolution operation with γ groups and weights ω iγ ∈ R C
Revised: The grouped convolution operation with γ groups and weights ω iγ ∈ R C

Original: j +1 ×
Revised: j +1 ×

Original: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.
Revised: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.

Original: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC j + (cid:107) , which only uses 1 γ of the input channels:
Revised: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC j + (cid:107) , which only uses 1 γ of the input channels:

Original: [Equation]
Revised: [Equation]

Original: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×
Revised: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×

Original: C j × s 2 j such that mask jm ( k, l, : , :)
Revised: C j × s 2 j such that mask jm ( k, l, : , :)

Original: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)
Revised: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)

Original: =
Revised: =

Original: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .
Revised: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .

Original: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) m =0 mask jm and the grouped convolution can therefore be rewritten as z j +1 = h
Revised: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) m =0 mask jm and the grouped convolution can therefore be rewritten as z j +1 = h

Original: j ⊛ (cid:0) ω j ◦ mask j (cid:1) , where ◦ is the Hadamard product.
Revised: j ⊛ (cid:0) ω j ◦ mask j (cid:1) , where ◦ is the Hadamard product.

Original: 2. B ACKGROUND ON D EEP E NSEMBLES
Revised: 2. B ACKGROUND ON D EEP E NSEMBLES

Original: For a classification problem, let us define a dataset D
Revised: For an image classification problem, let us define a dataset D

Original: = { x i , y i } |D| i =1 containing |D| pairs of samples x i  ∈ {+R C 
+}
Revised: = { x i , y i } |D| i =1 containing |D| pairs of samples x i = h 0 i ∈ {+R C 
+}

Original: R H j × W j and one-hot-encoded labels
Revised: × H 0 × W 0 and one-hot-encoded labels

Original: y i ∈
Revised: y i ∈

Original: R N C modeled as the realisation of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.
Revised: R N C modeled as the realization of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.

Original: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i
Revised: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i

Original: =
Revised: =

Original: f θ ( x i ) =
Revised: f θ ( x i ) =

Original: P ( Y = y
Revised: P ( Y = y

Original: i
Revised: i

Original: | X = x i ; θ ) .
Revised: | X = x i ; θ ) .

Original: This approach consists in considering the prediction ˆ y  as parameters of a Multinoulli distribution.
Revised: This approach consists in considering the prediction ˆ y i as parameters of a Multinoulli distribution.

Original: equivalent architectures 3,3x3,
Revised: equivalent architectures 3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Original: 72,1x1,144groups =
Revised: 72,1x1,144groups =

Original: Rearrange xxx3,3x3,
Revised: Rearrange xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Original: 72,1x1,144groups =
Revised: 72,1x1,144groups =

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: To improve the quality of both predictions and estimated uncertainties, as well as the detection of OOD samples, Lakshminarayanan et al.
Revised: To improve the quality of both predictions and estimated uncertainties, as well as the detection of OOD samples, Lakshminarayanan et al.

Original: (2017) propose to ensemble M randomly initialized
Revised: (2017) propose to ensemble M randomly initialized

Original: DNNs as a large predictor called Deep Ensembles.
Revised: DNNs as a large predictor called Deep Ensembles.

Original: These ensembles can be seen as a discrete approximation of the intractable Bayesian marginalization on the weights, according to Wilson &
Revised: These ensembles can be seen as a discrete approximation of the intractable Bayesian marginalization on the weights, according to Wilson &

Original: Izmailov (2020).
Revised: Izmailov (2020).

Original: If we note { θ m } M − 1 m =

Revised: If we note { θ m } M − 1 m =

Original: the set of trained weights for the M DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (3).
Revised: the set of trained weights for the M DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (3).

Original: [Equation]
Revised: [Equation]

Original:  P ACKED -E NSEMBLES
Revised:  P ACKED -E NSEMBLES

Original: This section describes how to train multiple subnetworks using grouped convolution efficiently.
Revised: This section describes how to train multiple subnetworks using grouped convolution efficiently.

Original: Then, we explain how our new architectures are equivalent to training several networks in parallel.
Revised: Then, we explain how our new architectures are equivalent to training several networks in parallel.

Original: 3. R EVISITING D EEP E NSEMBLES
Revised: 3. R EVISITING D EEP E NSEMBLES

Original: Although Deep Ensembles provide undisputed benefits, they also come with the significant drawback that the training time and the memory usage in inference increase linearly with the number of networks.
Revised: Although Deep Ensembles provide undisputed benefits, they also come with the significant drawback that the training time and the memory usage in inference increase linearly with the number of networks.

Original: To alleviate these problems, we propose assembling small subnetworks, which are essentially DNNs with fewer parameters.
Revised: To alleviate these problems, we propose assembling small subnetworks, which are essentially DNNs with fewer parameters.

Original: Moreover, while ensembles to this day have mostly been trained sequentially, we suggest leveraging grouped convolutions to massively accelerate their training and inference computations thanks to their smaller size.
Revised: Moreover, while ensembles to this day have mostly been trained sequentially, we suggest leveraging grouped convolutions to massively accelerate their training and inference computations thanks to their smaller size.

Original: The propagation of grouped convolutions with
Revised: The propagation of grouped convolutions with

Original: M groups, M being the number of subnetworks in the ensemble, ensures that the subnetworks are trained independently while dividing their encoding dimension by a factor M .
Revised: M groups, M being the number of subnetworks in the ensemble, ensures that the subnetworks are trained independently while dividing their encoding dimension by a factor M .

Original: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.
Revised: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.

Original: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate for the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as an expansion factor.
Revised: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate for the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as an expansion factor.

Original: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.
Revised: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.

Original: For an ensemble of M subnetworks,
Revised: For an ensemble of M subnetworks,

Original: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor α M and the inference of our ensemble is computed with the following formula:
Revised: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor α M and the inference of our ensemble is computed with the following formula, omitting the index i of the sample:

Original: [Equation]
Revised: [Equation]

Original: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×
Revised: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×

Original: s 2 j .
Revised: s 2 j .

Original: [Figure]
Revised: [Figure]

Original: m= m=
Revised: m= m=

Original: [Figure]
Revised: [Figure]

Original: In the following, we introduce another hyperparameter γ corresponding to the number of groups of each subnetwork of the Packed-Ensembles, creating another level of sparsity.
Revised: In the following, we introduce another hyperparameter γ corresponding to the number of groups of each subnetwork of the Packed-Ensembles, creating another level of sparsity.

Original: These groups are also called ”subgroups” and are applied to the different subnetworks.
Revised: These groups are also called ”subgroups” and are applied to the different subnetworks.

Original: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , with the hyperparameters in the parentheses.
Revised: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , with the hyperparameters in the parentheses.

Original: In this work, we consider a constant number of subgroups across the layers; therefore, γ divides αC j for all j .
Revised: In this work, we consider a constant number of subgroups across the layers; therefore, γ divides αC j for all j .

Original: 3. C OMPUTATIONAL COST
Revised: 3. C OMPUTATIONAL COST

Original: For a convolutional layer, the number of parameters involving C j input channels, C j +1 output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)
Revised: For a convolutional layer, the number of parameters involving C j input channels, C j +1 output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)

Original: [Equation]
Revised: [Equation]

Original: The same formula applies to dense layers as 1 × 1 convolutions.
Revised: The same formula applies to dense layers as 1 × 1 convolutions.

Original: Two cases emerge when the architectures of the subnetworks are fully convolutional or dense.
Revised: Two cases emerge when the architectures of the subnetworks are fully convolutional or dense.

Original: If α = √ M , the number of parameters in the ensemble equals the number of parameters in a single model.
Revised: If α = √ M , the number of parameters in the ensemble equals the number of parameters in a single model.

Original: With α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent in size to DE).
Revised: With α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent in size to DE).

Original: 3. I MPLEMENTATION DETAILS
Revised: 3. I MPLEMENTATION DETAILS

Original: We propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.
Revised: We propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.

Original: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) in Figure 3, with the parallel implementations (b) and (c) .
Revised: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) in Figure 3, with the parallel implementations (b) and (c) .

Original: Figure 3 summarizes different equivalent architectures for a simple ensemble of M =
Revised: Figure 3 summarizes different equivalent architectures for a simple ensemble of M =

Original: DNNs with three convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .
Revised: DNNs with three convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .

Original: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the rearrange operation).
Revised: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the rearrange operation).

Original: This yields a feature map h j , of size M × C i × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.
Revised: This yields a feature map h j , of size M × C j × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.

Original: One solution to keep the same batch size is to repeat the batch M times so that its size equals B after the rearrangement.
Revised: One solution to keep the same batch size is to repeat the batch M times so that its size equals B after the rearrangement.

Original: Using convolutions with M groups and γ subgroups per subnetwork, each feature map is convoluted separately by each subnetwork and yields its own independent output.
Revised: Using convolutions with M groups and γ subgroups per subnetwork, each feature map is convoluted separately by each subnetwork and yields its own independent output.

Original: Grouped convolutions are propagated until the end to ensure that gradients stay independent between subnetworks.
Revised: Grouped convolutions are propagated until the end to ensure that gradients stay independent between subnetworks.

Original: Other operations, such as Batch
Revised: Other operations, such as Batch

Original: Normalization (Ioffe & Szegedy, 2015), can be applied directly as long as they can be grouped or have independent actions on each channel.
Revised: Normalization (Ioffe & Szegedy, 2015), can be applied directly as long as they can be grouped or have independent actions on each channel.

Original: Figure 4a illustrates the mask used to code PackedEnsembles in the case where M = 2 .
Revised: Figure 4a illustrates the mask used to code PackedEnsembles in the case where M = 2 .

Original: Similarly, Figure 4b shows the mask with M = 2 and γ = 2 .
Revised: Similarly, Figure 4b shows the mask with M = 2 and γ = 2 .

Original: Finally, (b) and (c) are also equivalent.
Revised: Finally, (b) and (c) are also equivalent.

Original: It is indeed possible to replace the rearrange operation and the first grouped convolution with a standard convolution if the same images are to be provided simultaneously to all the subnetworks.
Revised: It is indeed possible to replace the rearrange operation and the first grouped convolution with a standard convolution if the same images are to be provided simultaneously to all the subnetworks.

Original: We confirm in Appendix F that this procedure is not detrimental to the ensemble’s performance, and we take advantage of this property to provide this final optimization and simplification.
Revised: We confirm in Appendix F that this procedure is not detrimental to the ensemble’s performance, and we take advantage of this property to provide this final optimization and simplification.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To validate the performance of our method, we conduct experiments on classification tasks and measure the influence of the parameters α and γ .
Revised: To validate the performance of our method, we conduct experiments on classification tasks and measure the influence of the parameters α and γ .

Original: Regression tasks are detailed in Appendix N.
Revised: Regression tasks are detailed in Appendix N.

Original: 4. D ATASETS AND ARCHITECTURES
Revised: 4. D ATASETS AND ARCHITECTURES

Original: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to tasks of different complexities.
Revised: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to tasks of different complexities.

Original: As we propose to replace a single model architecture with several subnetworks, we study the behavior of PE on various sizes architectures: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet2810 (Zagoruyko & Komodakis, 2016).
Revised: As we propose to replace a single model architecture with several subnetworks, we study the behavior of PE on various sizes architectures: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet2810 (Zagoruyko & Komodakis, 2016).

Original: We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and three other approximated ensembles from the literature: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020), and Masksembles (Durasov et al., 2021).
Revised: We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and three other approximated ensembles from the literature: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020), and Masksembles (Durasov et al., 2021).

Original: Second, we report our results for Packed-Ensembles on ImageNet (Deng et al., 2009), which we compare against all baselines.
Revised: Second, we report our results for Packed-Ensembles on ImageNet (Deng et al., 2009), which we compare against all baselines.

Original: We run experiments with ResNet-50 and ResNet-50x4.
Revised: We run experiments with ResNet-50 and ResNet-50x4.

Original: All training runs are started from scratch.
Revised: All training runs are started from scratch.

Original: 4.1. M ETRICS , OOD DATASETS , AND IMPLEMENTATION
Revised: 4.1. M ETRICS , OOD DATASETS , AND IMPLEMENTATION

Original: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and the Negative Log-Likelihood (NLL).
Revised: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and the Negative Log-Likelihood (NLL).

Original: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2 and measure the quality of the OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in %, similarly to
Revised: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2 and measure the quality of the OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in %, similarly to

Original: Hendrycks & Gimpel (2017).
Revised: Hendrycks & Gimpel (2017).

Original: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).
Revised: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).

Original: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble.
Revised: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble.

Original: We define the prediction confidence as this maximum value (also called maximum softmax probability).
Revised: We define the prediction confidence as this maximum value (also called maximum softmax probability).

Original: For OOD detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution and OOD data using the maximum softmax probability as criterion.
Revised: For OOD detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution and OOD data using the maximum softmax probability as criterion.

Original: We discuss the different OOD criteria in appendix E. For ImageNet, we use two out-ofdistribution datasets:
Revised: We discuss the different OOD criteria in appendix E. For ImageNet, we use two out-ofdistribution datasets:

Original: ImageNet-O (Hendrycks et al., 2021b) and Texture (Wang et al., 2022), and use the Mutual Information (MI) as a criterion for the ensembles techniques (see Appendix E for details on MI) and the maximum softmax probability for the single model and MIMO.
Revised: ImageNet-O (Hendrycks et al., 2021b) and Texture (Wang et al., 2022), and use the Mutual Information (MI) as a criterion for the ensembles techniques (see Appendix E for details on MI) and the maximum softmax probability for the single model and MIMO.

Original: To measure the robustness under distribution shift, we use ImageNet-R (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denoted rAcc, rECE, and rNLL on this dataset, respectively.
Revised: To measure the robustness under distribution shift, we use ImageNet-R (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denoted rAcc, rECE, and rNLL on this dataset, respectively.

Original: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.
Revised: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.

Original: Both are open-source Python frameworks.
Revised: Both are open-source Python frameworks.

Original: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.
Revised: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.

Original: Most training instances are completed on a single
Revised: Most training instances are completed on a single

Original: Nvidia RTX 3090 except for ImageNet, for which we use 2 to 8 Nvidia A100-80GB.
Revised: Nvidia RTX 3090 except for ImageNet, for which we use 2 to 8 Nvidia A100-80GB.

Original: 4.1. R ESULTS
Revised: 4.1. R ESULTS

Original: Table 1 presents the average performance for the classification task over five runs using the hyperparameters in Table 4.
Revised: Table 1 presents the average performance for the classification task over five runs using the hyperparameters in Table 4.

Original: We demonstrate that Packed-Ensembles, in the setting of α = 2 and γ = 2 , yields similar results to Deep Ensembles while having a lower memory cost than a single model.
Revised: We demonstrate that Packed-Ensembles, in the setting of α = 2 and γ = 2 , yields similar results to Deep Ensembles while having a lower memory cost than a single model.

Original: For CIFAR-10, the relative performance of PE compared to DE appears to increase as the original architecture becomes larger.
Revised: For CIFAR-10, the relative performance of PE compared to DE appears to increase as the original architecture becomes larger.

Original: When using ResNet-18, Packed-Ensembles matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.
Revised: When using ResNet-18, Packed-Ensembles matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.

Original: However, using
Revised: However, using

Original: ResNet-50, both models seem to perform similarly, and PE slightly outperforms DE in classification performance with WideResNet
Revised: ResNet-50, both models seem to perform similarly, and PE slightly outperforms DE in classification performance with WideResNet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5.19 MIMO ( ρ = 1 )
Revised: 5.19 MIMO ( ρ = 1 )

Original: C
Revised: C

Original: R 95.
Revised: R 95.

Original: 1.30 Masksembles
Revised: 1.30 Masksembles

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: 5.19 Packed-Ensembles
Revised: 5.19 Packed-Ensembles

Original: C
Revised: C

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1.00 Deep Ensembles
Revised: 1.00 Deep Ensembles

Original: C
Revised: C

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Single Model
Revised: Single Model

Original: [Table]
Revised: [Table]

Original: WR
Revised: WR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 95 BatchEnsemble
Revised: 95 BatchEnsemble

Original: C
Revised: C

Original: WR 95.
Revised: WR 95.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 23.81 MIMO
Revised: 23.81 MIMO

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: WR 94.
Revised: WR 94.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 96 Masksembles
Revised: 96 Masksembles

Original: [Table]
Revised: [Table]

Original: WR
Revised: WR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 23.82 Packed-Ensembles
Revised: 23.82 Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: WR 96.
Revised: WR 96.

Original: [Table]
Revised: [Table]

Original: 4.06 Deep Ensembles
Revised: 4.06 Deep Ensembles

Original: C
Revised: C

Original: WR 95.
Revised: WR 95.

Original: Single Model
Revised: Single Model

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: .56 BatchEnsemble
Revised: .56 BatchEnsemble

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: On CIFAR-100, Deep Ensembles outperforms Packed-Ensembles on ResNet-18.
Revised: On CIFAR-100, Deep Ensembles outperforms Packed-Ensembles on ResNet-18.

Original: However, we argue that ResNet-18 architecture need more representation capacity to be divided into subnetworks for CIFAR-100.
Revised: However, we argue that ResNet-18 architecture need more representation capacity to be divided into subnetworks for CIFAR-100.

Original: Indeed, when we look at the results of ResNet-50, we can see that PackedEnsembles has better results than Deep Ensembles.
Revised: Indeed, when we look at the results of ResNet-50, we can see that PackedEnsembles has better results than Deep Ensembles.

Original: This analysis demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.
Revised: This analysis demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.

Original: In Appendix D, we discuss the influence of the representation capacity.
Revised: In Appendix D, we discuss the influence of the representation capacity.

Original: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.
Revised: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.

Original: 4.1. S TUDY ON THE PARAMETERS α AND γ
Revised: 4.1. S TUDY ON THE PARAMETERS α AND γ

Original: Table 1 reports results for α = 2 and γ = 2 .
Revised: Table 1 reports results for α = 2 and γ = 2 .

Original: However, the optimal values of these hyperparameters depend on the balance between computational cost and performance.
Revised: However, the optimal values of these hyperparameters depend on the balance between computational cost and performance.

Original: To help users strike the best compromise, we propose Figures 6 and 7 in Appendix D, which illustrate the impact of changing α on the performance of Packed-Ensembles.
Revised: To help users strike the best compromise, we propose Figures 6 and 7 in Appendix D, which illustrate the impact of changing α on the performance of Packed-Ensembles.

Original:  D ISCUSSIONS
Revised:  D ISCUSSIONS

Original: We have shown that Packed-Ensembles has attractive properties, mainly by providing a similar quality of uncertainty quantification as Deep Ensembles while using a reduced architecture and
Revised: We have shown that Packed-Ensembles has attractive properties, mainly by providing a similar quality of uncertainty quantification as Deep Ensembles while using a reduced architecture and

Original: [Table]
Revised: [Table]

Original: computing cost.
Revised: computing cost.

Original: Several questions can be raised, and we conducted some studies - detailed in the
Revised: Several questions can be raised, and we conducted some studies - detailed in the

Original: Appendix sections - to provide possible answers.
Revised: Appendix sections - to provide possible answers.

Original: Discussion on the sparsity As described in section 3, one could interpret PE as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.
Revised: Discussion on the sparsity As described in section 3, one could interpret PE as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.

Original: In Appendix C, by using a simplified model, we propose a bound of the approximation error based on the Kullback-Leibler divergence between the DE and its pruned version.
Revised: In Appendix C, by using a simplified model, we propose a bound of the approximation error based on the Kullback-Leibler divergence between the DE and its pruned version.

Original: This bound depends on the density of ones in the mask p , and, more specifically, depends on p (1 − p ) and (1 − p ) 2 /p .
Revised: This bound depends on the density of ones in the mask p , and, more specifically, depends on p (1 − p ) and (1 − p ) 2 /p .

Original: By manipulating these terms, corresponding to modifying the number of subnetworks M , the number of groups γ , and the dilation factor α , we could theoretically control the approximation error.
Revised: By manipulating these terms, corresponding to modifying the number of subnetworks M , the number of groups γ , and the dilation factor α , we could theoretically control the approximation error.

Original: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two primary sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.
Revised: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two primary sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.

Original: A last source of stochasticity is introduced during training by the non-deterministic behavior of the backpropagation algorithms.
Revised: A last source of stochasticity is introduced during training by the non-deterministic behavior of the backpropagation algorithms.

Original: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.
Revised: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.

Original: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seems to emerge to predict the best combination.
Revised: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seems to emerge to predict the best combination.

Original: Specifically, we highlight that even the only use of non-deterministic algorithms introduces enough diversity between each subnetwork of the ensemble.
Revised: Specifically, we highlight that even the only use of non-deterministic algorithms introduces enough diversity between each subnetwork of the ensemble.

Original: Ablation study We perform ablation studies to assess the impact of the parameters M , α , and γ on the performance of Packed-Ensembles.
Revised: Ablation study We perform ablation studies to assess the impact of the parameters M , α , and γ on the performance of Packed-Ensembles.

Original: Appendix D provides in-depth details of this study.
Revised: Appendix D provides in-depth details of this study.

Original: No explicit behavior appears from the results we obtained.
Revised: No explicit behavior appears from the results we obtained.

Original: A trend shows that a higher number of subnetworks helps get better OOD detection, but the improvement in AUPR is not significant.
Revised: A trend shows that a higher number of subnetworks helps get better OOD detection, but the improvement in AUPR is not significant.

Original: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.
Revised: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.

Original: This translates into an expected lower number of operations.
Revised: This translates into an expected lower number of operations.

Original: A study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.
Revised: A study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.

Original: However, this also hints that the group-convolution speedup is not optimal despite the significant acceleration offered by 16-bits floating points.
Revised: However, this also hints that the group-convolution speedup is not optimal despite the significant acceleration offered by 16-bits floating points.

Original: OOD criteria The maximum softmax probability is often used as criterion for discriminating
Revised: OOD criteria The maximum softmax probability is often used as criterion for discriminating

Original: OOD elements.
Revised: OOD elements.

Original: However, this criterion is not unique, and others can be used, such as the Mutual
Revised: However, this criterion is not unique, and others can be used, such as the Mutual

Original: Information, the maximum logit, or the Shannon entropy of the mean prediction.
Revised: Information, the maximum logit, or the Shannon entropy of the mean prediction.

Original: Although no relationship is expected between this criterion and PE, we obtained different performances in OOD detection according to the selected criterion.
Revised: Although no relationship is expected between this criterion and PE, we obtained different performances in OOD detection according to the selected criterion.

Original: The results on CIFAR-100 are detailed in Appendix E and show that an approach based on the maximum logit seems to give the best results in detecting
Revised: The results on CIFAR-100 are detailed in Appendix E and show that an approach based on the maximum logit seems to give the best results in detecting

Original: OOD.
Revised: OOD.

Original: It should be noted that the notion of OOD depends on the training distribution.
Revised: It should be noted that the notion of OOD depends on the training distribution.

Original: Such a discussion does not necessarily generalize to all datasets.
Revised: Such a discussion does not necessarily generalize to all datasets.

Original: Indeed, preliminary results have shown that
Revised: Indeed, preliminary results have shown that

Original: Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.
Revised: Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Ensembles and uncertainty quantification.
Revised: Ensembles and uncertainty quantification.

Original: Bayesian Neural Networks (BNNs) (MacKay, 1992;
Revised: Bayesian Neural Networks (BNNs) (MacKay, 1992;

Original: Neal, 1995) are the cornerstone and primary source of inspiration for uncertainty quantification in deep learning.
Revised: Neal, 1995) are the cornerstone and primary source of inspiration for uncertainty quantification in deep learning.

Original: Despite the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain challenging to scale and train for large DNN architectures (Dusenberry et al., 2020).
Revised: Despite the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain challenging to scale and train for large DNN architectures (Dusenberry et al., 2020).

Original: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of
Revised: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of

Original: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).
Revised: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).

Original: DE are currently the best-performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: DE are currently the best-performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: Efficient ensembles.
Revised: Efficient ensembles.

Original: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost, have inspired a large cohort of approaches aiming to mitigate it.
Revised: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost, have inspired a large cohort of approaches aiming to mitigate it.

Original: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork-specific parameters trained in parallel.
Revised: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork-specific parameters trained in parallel.

Original: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.
Revised: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.

Original: A single network can be used in ensemble mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).
Revised: A single network can be used in ensemble mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).

Original: Liu et al.
Revised: Liu et al.

Original: (2022) leverage the sparse networks training algorithm of Mocanu et al.
Revised: (2022) leverage the sparse networks training algorithm of Mocanu et al.

Original: (2018) to produce ensembles of sparse networks.
Revised: (2018) to produce ensembles of sparse networks.

Original: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020), and by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).
Revised: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020), and by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).

Original: However, most of these approaches require multiple forward passes.
Revised: However, most of these approaches require multiple forward passes.

Original: Neural
Revised: Neural

Original: [Table]
Revised: [Table]

Original: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by design, relying on, e.g., channel shuffling (Zhang & Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.
Revised: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by design, relying on, e.g., channel shuffling (Zhang & Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.

Original: Some of the most popular architectures that leverage such models are SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b), and MobileNet-v3 (Howard et al., 2019).
Revised: Some of the most popular architectures that leverage such models are SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b), and MobileNet-v3 (Howard et al., 2019).

Original: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;
Revised: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;

Original: Louizos et al., 2018; Frankle & Carbin, 2018;
Revised: Louizos et al., 2018; Frankle & Carbin, 2018;

Original: Tartaglione et al., 2022).
Revised: Tartaglione et al., 2022).

Original: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency, also at train time Bragagnolo et al.
Revised: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency, also at train time Bragagnolo et al.

Original: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.
Revised: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.

Original: Grouped convolutions.
Revised: Grouped convolutions.

Original: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.
Revised: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.

Original: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.
Revised: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.

Original: Xie et al.
Revised: Xie et al.

Original: (2017) demonstrate that using grouped convolutions leads to accuracy improvements and model complexity reduction.
Revised: (2017) demonstrate that using grouped convolutions leads to accuracy improvements and model complexity reduction.

Original: So far, grouped convolutions have been used primarily for computational efficiency but also to compute multiple output branches in parallel (Chen & Shrivastava, 2020).
Revised: So far, grouped convolutions have been used primarily for computational efficiency but also to compute multiple output branches in parallel (Chen & Shrivastava, 2020).

Original: PE re-purpose them to delineate multiple subnetworks within a network and efficiently train an ensemble of such subnetworks.
Revised: PE re-purpose them to delineate multiple subnetworks within a network and efficiently train an ensemble of such subnetworks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a new ensemble framework, Packed-Ensembles, that can approximate Deep Ensembles in terms of uncertainty quantification and accuracy.
Revised: We propose a new ensemble framework, Packed-Ensembles, that can approximate Deep Ensembles in terms of uncertainty quantification and accuracy.

Original: Our research provides several new findings.
Revised: Our research provides several new findings.

Original: First, we show that small independent neural networks can be as effective as large, deep neural networks when used in ensembles.
Revised: First, we show that small independent neural networks can be as effective as large, deep neural networks when used in ensembles.

Original: Secondly, we demonstrate that not all sources of diversity are essential for improving ensemble diversity.
Revised: Secondly, we demonstrate that not all sources of diversity are essential for improving ensemble diversity.

Original: Thirdly, we show that Packed-Ensembles are more stable than single DNNs.
Revised: Thirdly, we show that Packed-Ensembles are more stable than single DNNs.

Original: Fourthly, we highlight that there is a trade-off between accuracy and the number of parameters, and Packed-Ensembles enables us to create flexible and efficient ensembles.
Revised: Fourthly, we highlight that there is a trade-off between accuracy and the number of parameters, and Packed-Ensembles enables us to create flexible and efficient ensembles.

Original: In the future, we intend to explore Packed-Ensembles for more complex downstream tasks.
Revised: In the future, we intend to explore Packed-Ensembles for more complex downstream tasks.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: Alongside this paper, we provide the source code of Packed-Ensembles layers.
Revised: Alongside this paper, we provide the source code of Packed-Ensembles layers.

Original: Additionally, we have created two notebooks demonstrating how to train ResNet-50-based Packed-Ensembles using public datasets such as CIFAR-10 and CIFAR-100.
Revised: Additionally, we have created two notebooks demonstrating how to train ResNet-50-based Packed-Ensembles using public datasets such as CIFAR-10 and CIFAR-100.

Original: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.
Revised: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.

Original: Furthermore, it should be noted that the source code contains two PyTorch Module classes to produce Packed-Ensembles efficiently.
Revised: Furthermore, it should be noted that the source code contains two PyTorch Module classes to produce Packed-Ensembles efficiently.

Original: A readme file at the root of the project details how to install and run experiments.
Revised: A readme file at the root of the project details how to install and run experiments.

Original: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).
Revised: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).

Original: To further promote accessibility, we have created an open-source pip-installable PyTorch package, torch-uncertainty, that includes Packed-Ensembles layers.
Revised: To further promote accessibility, we have created an open-source pip-installable PyTorch package, torch-uncertainty, that includes Packed-Ensembles layers.

Original: With these resources, we hope to encourage the broader research community to engage with and build upon our work.
Revised: With these resources, we hope to encourage the broader research community to engage with and build upon our work.

Original:  E THICS
Revised:  E THICS

Original: The purpose of this paper is to propose a new method for better estimations of uncertainty for deep-learning-based models.
Revised: The purpose of this paper is to propose a new method for better estimations of uncertainty for deep-learning-based models.

Original: Nevertheless, we acknowledge their limitations, which could become particularly concerning when applied to safety-critical systems.
Revised: Nevertheless, we acknowledge their limitations, which could become particularly concerning when applied to safety-critical systems.

Original: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.
Revised: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.

Original: We show the limitations of our approach in several experiments.
Revised: We show the limitations of our approach in several experiments.

Original: Many more validation and verification steps would be crucial before considering its real-world implementation to ensure robustness to various unknown situations, including corner cases, adversarial attacks, and potential biases.
Revised: Many more validation and verification steps would be crucial before considering its real-world implementation to ensure robustness to various unknown situations, including corner cases, adversarial attacks, and potential biases.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported by AID Project ACoCaTherm and Hi!Paris.
Revised: This work was supported by AID Project ACoCaTherm and Hi!Paris.

Original: This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-AD011011970R1) and (Grant 2022AD011011970R2).
Revised: This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-AD011011970R1) and (Grant 2022AD011011970R2).

Original: T ABLE OF C ONTENTS - S UPPLEMENTARY M ATERIAL A Notations
Revised: T ABLE OF C ONTENTS - S UPPLEMENTARY M ATERIAL A Notations

Original: B Implementation details
Revised: B Implementation details

Original: C Discussion on the sparsity
Revised: C Discussion on the sparsity

Original: D Ablation study
Revised: D Ablation study

Original: E Discussion about OOD criteria
Revised: E Discussion about OOD criteria

Original: F Discussion about the sources of stochasticity
Revised: F Discussion about the sources of stochasticity

Original: G Discussion about the subnetworks
Revised: G Discussion about the subnetworks

Original: H Discussion about the training velocity
Revised: H Discussion about the training velocity

Original: I
Revised: I

Original: Distribution shift
Revised: Distribution shift

Original: J Stabilization of the performance
Revised: J Stabilization of the performance

Original: K On the equivalence between sequential training and Packed-Ensembles
Revised: K On the equivalence between sequential training and Packed-Ensembles

Original: L Using groups is not sufficient to equal Packed-Ensembles
Revised: L Using groups is not sufficient to equal Packed-Ensembles

Original: M Efficiency of the networks trained on ImageNet
Revised: M Efficiency of the networks trained on ImageNet

Original: N Regression
Revised: N Regression

Original: A N OTATIONS
Revised: A N OTATIONS

Original: We summarize the main notations used in the paper in Table
Revised: We summarize the main notations used in the paper in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: of
Revised: of

Original: [Table]
Revised: [Table]

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: data samples and the corresponding labels
Revised: data samples and the corresponding labels

Original: [Table]
Revised: [Table]

Original: The index of the current layer, the current subnetwork, and the number of layers
Revised: The index of the current layer, the current subnetwork, and the number of layers

Original: [Table]
Revised: [Table]

Original: The preactivation feature map
Revised: The preactivation feature map

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: layer j
Revised: layer j

Original: [Table]
Revised: [Table]

Original: The activation function (considered constant throughout the network)
Revised: The activation function (considered constant throughout the network)

Original: [Table]
Revised: [Table]

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: channels
Revised: channels

Original: [Table]
Revised: [Table]

Original: the feature maps
Revised: the feature maps

Original: [Table]
Revised: [Table]

Original: layer
Revised: layer

Original: [Table]
Revised: [Table]

Original: The number of parameters of layer j
Revised: The number of parameters of layer j

Original: [Table]
Revised: [Table]

Original: The batch size of the training procedure
Revised: The batch size of the training procedure

Original: [Table]
Revised: [Table]

Original: The mask corresponding to the layer j of the subnetwork m
Revised: The mask corresponding to the layer j of the subnetwork m

Original: [Table]
Revised: [Table]

Original: The floor function
Revised: The floor function

Original: [Table]
Revised: [Table]

Original: The 2D cross-correlation, the convolution, and the Hadamard product
Revised: The 2D cross-correlation, the convolution, and the Hadamard product

Original: [Table]
Revised: [Table]

Original: The size of the kernel of the layer j
Revised: The size of the kernel of the layer j

Original: [Table]
Revised: [Table]

Original: The number of subnetworks in an ensemble
Revised: The number of subnetworks in an ensemble

Original: [Table]
Revised: [Table]

Original: The prediction of the subnetwork number m concerning the input x
Revised: The prediction of the subnetwork number m concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The prediction of the ensemble concerning the input x
Revised: The prediction of the ensemble concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The width-augmentation factor of Packed-Ensembles
Revised: The width-augmentation factor of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The number of subgroups of Packed-Ensembles
Revised: The number of subgroups of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The set of weights of the subnetwork m with a width factor α
Revised: The set of weights of the subnetwork m with a width factor α

Original: [Table]
Revised: [Table]

Original: The weights of layer j with γ groups and a width factor α
Revised: The weights of layer j with γ groups and a width factor α

Original: [Table]
Revised: [Table]

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: General Considerations.
Revised: General Considerations.

Original: Table 4 summarizes all the hyperparameters used in the paper for
Revised: Table 4 summarizes all the hyperparameters used in the paper for

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: In all cases, we use SGD combined with a multistep-learning-rate scheduler multiplying the rate by γ -lr at each milestone.
Revised: In all cases, we use SGD combined with a multistep-learning-rate scheduler multiplying the rate by γ -lr at each milestone.

Original: Note that BatchEnsemble based on ResNet50 uses a lower learning rate of 0.08 instead of 0.1 for stability.
Revised: Note that BatchEnsemble based on ResNet50 uses a lower learning rate of 0.08 instead of 0.1 for stability.

Original: The ”Medium” data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0 .
Revised: The ”Medium” data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0 .

Original: 5 switch probability and using timm’s augmentation classes (Wightman, 2019), with coefficients respectively 0.5 and 0.2.
Revised: 5 switch probability and using timm’s augmentation classes (Wightman, 2019), with coefficients respectively 0.5 and 0.2.

Original: In this case, we also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1.
Revised: In this case, we also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1.

Original: To ensure that the layers convey sufficient information and are not weakened by groups, we have set a constant minimum number of channels per group to 64 for all experiments presented in the paper.
Revised: To ensure that the layers convey sufficient information and are not weakened by groups, we have set a constant minimum number of channels per group to 64 for all experiments presented in the paper.

Original: If the number of channels per group is lower than this threshold, γ is reduced.
Revised: If the number of channels per group is lower than this threshold, γ is reduced.

Original: Moreover, we do not apply subgroups (parameterized by γ ) on the first layer of the network, nor on the first layer of
Revised: Moreover, we do not apply subgroups (parameterized by γ ) on the first layer of the network, nor on the first layer of

Original: ResNet’s blocks.
Revised: ResNet’s blocks.

Original: Experiments in which this minimum number of channels could play a significant role and bring confusion are not presented (see, for instance, PE-( 1 , 4 , 4 ) in Table 5).
Revised: Experiments in which this minimum number of channels could play a significant role and bring confusion are not presented (see, for instance, PE-( 1 , 4 , 4 ) in Table 5).

Original: For ImageNet, we use the A3 procedure from Wightman et al.
Revised: For ImageNet, we use the A3 procedure from Wightman et al.

Original: (2021) for all models.
Revised: (2021) for all models.

Original: Training with the exact A3 procedure was not always possible.
Revised: Training with the exact A3 procedure was not always possible.

Original: Refer to the specific subsection for more details.
Revised: Refer to the specific subsection for more details.

Original: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).
Revised: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).

Original: We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).
Revised: We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).

Original: Masksembles.
Revised: Masksembles.

Original: We use the code proposed by (Durasov et al., 2021) 3 .
Revised: We use the code proposed by (Durasov et al., 2021) 3 .

Original: We modified the mask generation function using binary search, as proposed by the authors since it was unable to build masks for ResNet50x4.
Revised: We modified the mask generation function using binary search, as proposed by the authors since it was unable to build masks for ResNet50x4.

Original: We note that the code implies performing batch repeats at the start of the forward passes.
Revised: We note that the code implies performing batch repeats at the start of the forward passes.

Original: All the results regarding this technique are therefore computed with this specification.
Revised: All the results regarding this technique are therefore computed with this specification.

Original: The
Revised: The

Original: ResNet implementations are built using Masksemble2D layers with M = 4 and a scale factor ofafter each convolution.
Revised: ResNet implementations are built using Masksemble2D layers with M = 4 and a scale factor ofafter each convolution.

Original: BatchEnsemble.
Revised: BatchEnsemble.

Original: For BatchEnsemble, we use two different values for weight decay: table 4 provides the weight decay corresponding to the shared weights but we don’t apply weight decay to the vectors S and R (which generate the rank-1 matrices).
Revised: For BatchEnsemble, we use two different values for weight decay: table 4 provides the weight decay corresponding to the shared weights but we don’t apply weight decay to the vectors S and R (which generate the rank-1 matrices).

Original: ImageNet.
Revised: ImageNet.

Original: The batch size of Masksembles ResNet-50x4 is reduced to 1120 because of memory constraints.
Revised: The batch size of Masksembles ResNet-50x4 is reduced to 1120 because of memory constraints.

Original: Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence.
Revised: Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence.

Original: C
Revised: C

Original: D ISCUSSION ON THE SPARSITY
Revised: D ISCUSSION ON THE SPARSITY

Original: In this section, we estimate the expected distance between a dense, fully-connected layer and a sparse one.
Revised: In this section, we estimate the expected distance between a dense, fully-connected layer and a sparse one.

Original: For simplicity, we are here assuming to operate with a fully-connected layer.
Revised: For simplicity, we are here assuming to operate with a fully-connected layer.

Original: First, let us write our first proposition:
Revised: First, let us write our first proposition:

Original: Proposition C.1.
Revised: Proposition C.1.

Original: Given a fully connected layer j + 1 defined by:
Revised: Given a fully connected layer j + 1 defined by:

Original: [Equation]
Revised: [Equation]

Original: and its approximation defined by:
Revised: and its approximation defined by:

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix, and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:
Revised: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix, and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜
Revised: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜

Original: z
Revised: z

Original: j +1 ( c ) .
Revised: j +1 ( c ) .

Original: A plot for (7) is provided in Figure 5.
Revised: A plot for (7) is provided in Figure 5.

Original: Proof.
Revised: Proof.

Original: To prove Prop.
Revised: To prove Prop.

Original: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,
Revised: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,

Original: z j +1 will be as well gaussian-distributed.
Revised: z j +1 will be as well gaussian-distributed.

Original: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:
Revised: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:

Original: [Equation]
Revised: [Equation]

Original: and the variance is:
Revised: and the variance is:

Original: [Equation]
Revised: [Equation]

Original: If we assume Σ( i, k ) = 0 ∀ i ̸
Revised: If we assume Σ( i, k ) = 0 ∀ i ̸

Original: = k , (9) simplifies into:
Revised: = k , (9) simplifies into:

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the case with the mask, similarly as presented at the end of section 2.1:
Revised: Let us now consider the case with the mask, similarly as presented at the end of section 2.1:

Original: [Equation]
Revised: [Equation]

Original: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).
Revised: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).

Original: In the limit of large C j , we know that ˜
Revised: In the limit of large C j , we know that ˜

Original: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:
Revised: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: In order to assess the dissimilarity between z and ˜ z
Revised: In order to assess the dissimilarity between z and ˜ z

Original: [Equation]
Revised: [Equation]

Original: we can write the Kullback–Leibler divergence:
Revised: we can write the Kullback–Leibler divergence:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Straightforwardly we can write the inequality:
Revised: Straightforwardly we can write the inequality:

Original: [Equation]
Revised: [Equation]

Original: According to (15) we can write:
Revised: According to (15) we can write:

Original: [Equation]
Revised: [Equation]

Original: Since we know that
Revised: Since we know that

Original: [Equation]
Revised: [Equation]

Original: we can also write:
Revised: we can also write:

Original: [Equation]
Revised: [Equation]

Original: Finally, according to: (14)
Revised: Finally, according to: (14)

Original: [Equation]
Revised: [Equation]

Original: finding back (7)
Revised: finding back (7)

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D A BLATION STUDY
Revised: D A BLATION STUDY

Original: Our algorithm mainly depends on three hyperparameters.
Revised: Our algorithm mainly depends on three hyperparameters.

Original: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.
Revised: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.

Original: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.
Revised: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.

Original: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performance, in terms of accuracy and AUPR.
Revised: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performance, in terms of accuracy and AUPR.

Original: We also note that the results are stable with γ .
Revised: We also note that the results are stable with γ .

Original: Moreover, the resulting accuracy tends to increase with α until it reaches a plateau.
Revised: Moreover, the resulting accuracy tends to increase with α until it reaches a plateau.

Original: These statements are confirmed by the results in Table 5.
Revised: These statements are confirmed by the results in Table 5.

Original: E D ISCUSSION ABOUT OOD CRITERIA
Revised: E D ISCUSSION ABOUT OOD CRITERIA

Original: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.
Revised: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.

Original: Similarly to Bayesian Neural Network, one can take the softmax outputs of posterior predictive distribution, which define the MSP = max y i { P ( y i | x , D ) } .
Revised: Similarly to Bayesian Neural Network, one can take the softmax outputs of posterior predictive distribution, which define the MSP = max y i { P ( y i | x , D ) } .

Original: The MSP can also be used for classical DNN, yet we use the conditional likelihood instead of the posterior distribution in this case.
Revised: The MSP can also be used for classical DNN, yet we use the conditional likelihood instead of the posterior distribution in this case.

Original: One can also use the Maximum Logit (ML) and the entropy of the posterior predictive distribution as uncertainty criteria, which is defined by Ent.
Revised: One can also use the Maximum Logit (ML) and the entropy of the posterior predictive distribution as uncertainty criteria, which is defined by Ent.

Original: =
Revised: =

Original: H ( P ( y i | x , D )) with H being the entropy function.
Revised: H ( P ( y i | x , D )) with H being the entropy function.

Original: Another metric is the mutual information between two random variables, which is defined by: MI =
Revised: Another metric is the mutual information between two random variables, which is defined by: MI =

Original: H ( P ( y i | x , D )) − 1 M M − 1 (cid:80) m =
Revised: H ( P ( y i | x , D )) − 1 M M − 1 (cid:80) m =

Original: 0 H ( P ( y | θ α,m , x )) .
Revised: 0 H ( P ( y | θ α,m , x )) .

Original: It represents a measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.
Revised: It represents a measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The last metric, used in active learning, is the variation ratio (Beluch et al., 2018), which measures the dispersion of a nominal variable and is calculated as the proportion of predicted class labels that are not the modal class prediction.
Revised: The last metric, used in active learning, is the variation ratio (Beluch et al., 2018), which measures the dispersion of a nominal variable and is calculated as the proportion of predicted class labels that are not the modal class prediction.

Original: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.
Revised: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: the results for the different metrics are reported.
Revised: the results for the different metrics are reported.

Original: We note that ML seems to be the best metric to detect OOD.
Revised: We note that ML seems to be the best metric to detect OOD.

Original: This metric is followed by Ent.
Revised: This metric is followed by Ent.

Original: and then MI.
Revised: and then MI.

Original: Note that v , widely used in active learning, does not seem effective in detecting OOD samples.
Revised: Note that v , widely used in active learning, does not seem effective in detecting OOD samples.

Original: This shows us that it is essential to use a good criterion in addition to good ensembling.
Revised: This shows us that it is essential to use a good criterion in addition to good ensembling.

Original: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY
Revised: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY

Original: As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection.
Revised: As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection.

Original: Three primary sources can induce weight diversity, and therefore diversity in the function space, during the training.
Revised: Three primary sources can induce weight diversity, and therefore diversity in the function space, during the training.

Original: These sources are the initialization of the weights, the composition of the batches, and the use of non-deterministic backpropagation algorithms 4 .
Revised: These sources are the initialization of the weights, the composition of the batches, and the use of non-deterministic backpropagation algorithms 4 .

Original: On Table 7, we measure the performance and diversity of PackedEnsembles trained on CIFAR-100.
Revised: On Table 7, we measure the performance and diversity of PackedEnsembles trained on CIFAR-100.

Original: The quantity of diversity is measured by the mutual information and is twofold: we compute the in-distribution mutual information (ID MI ) on the test set of
Revised: The quantity of diversity is measured by the mutual information and is twofold: we compute the in-distribution mutual information (ID MI ) on the test set of

Original: CIFAR-100 and the OOD mutual information (OOD MI ) on SVHN.
Revised: CIFAR-100 and the OOD mutual information (OOD MI ) on SVHN.

Original: Concerning the performance, we compute the accuracy, ECE, and AUPR, which are proxies of the quality of this diversity.
Revised: Concerning the performance, we compute the accuracy, ECE, and AUPR, which are proxies of the quality of this diversity.

Original: Results of Table 7 lead to several takeaways.
Revised: Results of Table 7 lead to several takeaways.

Original: First, they hint that there is no clear best set of trivial sources of stochasticity.
Revised: First, they hint that there is no clear best set of trivial sources of stochasticity.

Original: Except for the first (and greyed) line, which corresponds to ensembling completely identical networks (the training being totally deterministic, which the null MI confirms), the results seem equivalent in diversity (via mutual information) and ID/OOD performance.
Revised: Except for the first (and greyed) line, which corresponds to ensembling completely identical networks (the training being totally deterministic, which the null MI confirms), the results seem equivalent in diversity (via mutual information) and ID/OOD performance.

Original: Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity.
Revised: Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity.

Original: It was noted that this effect does not always happen depending on the selected architecture and the precision used ( float16 , or float32 ).
Revised: It was noted that this effect does not always happen depending on the selected architecture and the precision used ( float16 , or float32 ).

Original: Given that there is no emerging best set of stochasticity, we use the faster non-deterministic backpropagation algorithms and different initializations to ensure enough stochasticity and for programming convenience.
Revised: Given that there is no emerging best set of stochasticity, we use the faster non-deterministic backpropagation algorithms and different initializations to ensure enough stochasticity and for programming convenience.

Original: [Table]
Revised: [Table]

Original: G D ISCUSSION ABOUT THE SUBNETWORKS
Revised: G D ISCUSSION ABOUT THE SUBNETWORKS

Original: The width and depth of deep neural networks are crucial research topics, and researchers strive to determine the best approaches for increasing the depth of DNNs, which can lead to improved accuracy.
Revised: The width and depth of deep neural networks are crucial research topics, and researchers strive to determine the best approaches for increasing the depth of DNNs, which can lead to improved accuracy.

Original: According to Nguyen et al.
Revised: According to Nguyen et al.

Original: (2020), the width and depth of a DNN are connected with its capacity to learn block structures, which can improve accuracy.
Revised: (2020), the width and depth of a DNN are connected with its capacity to learn block structures, which can improve accuracy.

Original: Therefore, the model’s capacity may decrease if the width is divided.
Revised: Therefore, the model’s capacity may decrease if the width is divided.

Original: Deep neural networks are heavily over-parameterized, as stated by the lottery ticket hypothesis (Frankle & Carbin, 2018).
Revised: Deep neural networks are heavily over-parameterized, as stated by the lottery ticket hypothesis (Frankle & Carbin, 2018).

Original: It suggests that up to 80% of neurons can be removed without significant loss of performance.
Revised: It suggests that up to 80% of neurons can be removed without significant loss of performance.

Original: The MIMO approach builds on this assumption by allowing multiple networks to be trained simultaneously
Revised: The MIMO approach builds on this assumption by allowing multiple networks to be trained simultaneously

Original: [Table]
Revised: [Table]

Original: and neurons may be used by several subnetworks.
Revised: and neurons may be used by several subnetworks.

Original: In our work, however, we assign each neuron to a specific DNN in the ensemble, guaranteeing their independence.
Revised: In our work, however, we assign each neuron to a specific DNN in the ensemble, guaranteeing their independence.

Original: This way, the DNNs can learn independent representations.
Revised: This way, the DNNs can learn independent representations.

Original: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.
Revised: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.

Original: Although the decomposition may seem crude, it facilitates better parallelization of
Revised: Although the decomposition may seem crude, it facilitates better parallelization of

Original: Packed-Ensembles during training and inference.
Revised: Packed-Ensembles during training and inference.

Original: To address the problem of not having sufficiently wide subnetworks, we added a hyperparameterα - to increase the width of subnetworks.
Revised: To address the problem of not having sufficiently wide subnetworks, we added a hyperparameterα - to increase the width of subnetworks.

Original: In Figure 8, we explore the impact of subnetwork width.
Revised: In Figure 8, we explore the impact of subnetwork width.

Original: Our observations reveal that the accuracy of the DNN increases with the width while the AUPR remains relatively constant.
Revised: Our observations reveal that the accuracy of the DNN increases with the width while the AUPR remains relatively constant.

Original: This finding suggests that the α is paramount in maintaining a balance in the DNN’s width.
Revised: This finding suggests that the α is paramount in maintaining a balance in the DNN’s width.

Original: We also note that reducing the width of the DNN does not significantly impact its accuracy.
Revised: We also note that reducing the width of the DNN does not significantly impact its accuracy.

Original: Hence, our decision to split the width of the DNN to create multiple subnetworks is justified since the uncertainty quantification remains unaltered, and the accuracy is not significantly compromised.
Revised: Hence, our decision to split the width of the DNN to create multiple subnetworks is justified since the uncertainty quantification remains unaltered, and the accuracy is not significantly compromised.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In addition
Revised: In addition

Original: [Table]
Revised: [Table]

Original: α provides an additional degree of freedom to our ensemble
Revised: α provides an additional degree of freedom to our ensemble

Original: [Table]
Revised: [Table]

Original: enabling it to enhance its accuracy.
Revised: enabling it to enhance its accuracy.

Original: This is a significant advantage
Revised: This is a significant advantage

Original: [Table]
Revised: [Table]

Original: as it allows us to further balance the performance of the ensemble, which can lead to more accurate predictions - and the number of parameters linked to its computational cost.
Revised: as it allows us to further balance the performance of the ensemble, which can lead to more accurate predictions - and the number of parameters linked to its computational cost.

Original: H D ISCUSSION ABOUT THE TRAINING VELOCITY
Revised: H D ISCUSSION ABOUT THE TRAINING VELOCITY

Original: Our experiments show that grouped convolutions are not as fast as they could theoretically be, and confirm the statements made by many PyTorch and TensorFlow users 5 .
Revised: Our experiments show that grouped convolutions are not as fast as they could theoretically be, and confirm the statements made by many PyTorch and TensorFlow users 5 .

Original: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed
Revised: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed

Original: Precision (AMP) and cuDNN benchmark flags when training a Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.
Revised: Precision (AMP) and cuDNN benchmark flags when training a Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.

Original: AMP also divides the VRAM usage by two while yielding equally good results.
Revised: AMP also divides the VRAM usage by two while yielding equally good results.

Original: Future improvements of PyTorch grouped convolutions should help
Revised: Future improvements of PyTorch grouped convolutions should help

Original: Packed-Ensembles develop its full potential, increasing its current assets.
Revised: Packed-Ensembles develop its full potential, increasing its current assets.

Original: We note in Table 8 that using float16 , Packed-Ensembles is only 1 .
Revised: We note in Table 8 that using float16 , Packed-Ensembles is only 1 .

Original: 6 × slower than the single model during inference.
Revised: 6 × slower than the single model during inference.

Original: Furthermore, Packed-Ensembles is only 2 .
Revised: Furthermore, Packed-Ensembles is only 2 .

Original: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.
Revised: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.

Original: I D ISTRIBUTION SHIFT
Revised: I D ISTRIBUTION SHIFT

Original: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.
Revised: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.

Original: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are five levels of perturbations called ”severity”, from one, the weakest, to five, the strongest.
Revised: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are five levels of perturbations called ”severity”, from one, the weakest, to five, the strongest.

Original: In real-world scenarios,
Revised: In real-world scenarios,

Original: [Figure]
Revised: [Figure]

Original: distributional shift is crucial, as explained by (Ovadia et al., 2019), and it is critical to study how much a model prediction shifts from the original training data distribution.
Revised: distributional shift is crucial, as explained by (Ovadia et al., 2019), and it is critical to study how much a model prediction shifts from the original training data distribution.

Original: Thanks to Figure 9, we notice that Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.
Revised: Thanks to Figure 9, we notice that Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.

Original: J S TABILIZATION OF THE PERFORMANCE
Revised: J S TABILIZATION OF THE PERFORMANCE

Original: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute the variance.
Revised: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute the variance.

Original: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .
Revised: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .

Original: Ensemble strategies shrink the standard variation to 0 .
Revised: Ensemble strategies shrink the standard variation to 0 .

Original: 43% for Deep Ensembles and 0 .
Revised: 43% for Deep Ensembles and 0 .

Original: 19% for Packed-Ensembles.
Revised: 19% for Packed-Ensembles.

Original: Thus it seems that Packed-Ensembles makes DNN predictions more stable in addition to improving accuracy and uncertainty quantification.
Revised: Thus it seems that Packed-Ensembles makes DNN predictions more stable in addition to improving accuracy and uncertainty quantification.

Original: This result is interesting as it appears to contradict Neal et al.
Revised: This result is interesting as it appears to contradict Neal et al.

Original: (2019), who claim that wider DNNs have a smaller variance.
Revised: (2019), who claim that wider DNNs have a smaller variance.

Original: This stability might come from the ensembling.
Revised: This stability might come from the ensembling.

Original: K O N THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES
Revised: K O N THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES

Original: The sequential training of Deep Ensembles differs significantly from the training procedure of
Revised: The sequential training of Deep Ensembles differs significantly from the training procedure of

Original: Packed-Ensembles.
Revised: Packed-Ensembles.

Original: The main differences lie in the subnetworks’ batch composition and the best models’ selection.
Revised: The main differences lie in the subnetworks’ batch composition and the best models’ selection.

Original: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.
Revised: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.

Original: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see
Revised: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see

Original: Appendix F for more details).
Revised: Appendix F for more details).

Original: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.
Revised: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.

Original: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.
Revised: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.

Original: L U SING GROUPS IS NOT SUFFICIENT TO EQUAL P ACKED -E NSEMBLES
Revised: L U SING GROUPS IS NOT SUFFICIENT TO EQUAL P ACKED -E NSEMBLES

Original: To make sure that the use of groups cannot simply explain our results, we compare PackedEnsembles to a single ResNeXt-50 (32 × 4d) (Xie et al., 2017) in Table 9.
Revised: To make sure that the use of groups cannot simply explain our results, we compare PackedEnsembles to a single ResNeXt-50 (32 × 4d) (Xie et al., 2017) in Table 9.

Original: ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of each
Revised: ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of each

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: block, which are therefore not independent.
Revised: block, which are therefore not independent.

Original: We keep the same training optimization procedures and data-augmentation strategies detailed in Appendix B.
Revised: We keep the same training optimization procedures and data-augmentation strategies detailed in Appendix B.

Original: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET
Revised: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET

Original: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.
Revised: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.

Original: PE-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.
Revised: PE-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.

Original: N R EGRESSION
Revised: N R EGRESSION

Original: To generalize our work, we propose to study regression tasks.
Revised: To generalize our work, we propose to study regression tasks.

Original: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.
Revised: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.

Original: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (20) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).
Revised: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (20) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).

Original: One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance.
Revised: One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance.

Original: The ensemble’s mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ σ θ ( x i ) 

Revised: The ensemble’s mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ σ θ ( x i )

Original: = M − 1 (cid:80) m (cid:0) σ θ m ( x i ) 2 + µ θ m ( x i ) 2 (cid:1) − ¯ µ θ ( x i ) (Lakshminarayanan et al., 2017).
Revised: = M − 1 (cid:80) m (cid:0) σ θ m ( x i ) 2 + µ θ m ( x i ) 2 (cid:1) − ¯ µ θ ( x i ) (Lakshminarayanan et al., 2017).

Original: [Equation]
Revised: [Equation]

Original: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.
Revised: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.

Original: The subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, containing 400 neurons for the more extensive Protein dataset and 200 for the others, and a ReLU non-linearity.
Revised: The subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, containing 400 neurons for the more extensive Protein dataset and 200 for the others, and a ReLU non-linearity.

Original: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.
Revised: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.

Original: [Table]
Revised: [Table]