Original: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION
Revised: P ACKED -E NSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Ensembles (DE) are a prominent approach  achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-ofdistribution detection.
Revised: Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation and out-of-distribution detection.

Original: However, hardware limitations of real-world systems constrain to smaller ensembles and lower capacity networks, significantly deteriorating their performance and properties.
Revised: However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties.

Original: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.
Revised: We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space.

Original: We leverage grouped convolutions to parallelize the ensemble into a single common backbone and forward pass to improve training and inference speeds.
Revised: We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds.

Original: PE is designed to work under the memory budget of asingle standard neural network.
Revised: PE is designed to operate within the memory limits of a standard neural network.

Original: Through extensive studies we show that PE faithfully preserve the properties of DE, e.g., diversity, and match their performance in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.
Revised: Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity and performs equally well in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.

Revised: We make our code available at github.com/ENSTAU2IS/torch-uncertainty

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose particularly high reliability and performance requirements over a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.
Revised: Real-world safety-critical machine learning decision systems such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017) impose exceptionally high reliability and performance requirements across a broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources.

Original: Although their performanceacross all key dimensions has been dramatically improved in the last years, vanilla Deep Neural Networks (DNNs) still have several shortcomings, notablythe overconfidence on both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).
Revised: Despite significant improvements in performance in recent years, vanilla Deep Neural Networks (DNNs) still exhibit several shortcomings, notably overconfidence in both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019).

Original: Deep Ensembles(DE) (Lakshminarayanan et al., 2017) arise as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.
Revised: Deep Ensembles (Lakshminarayanan et al., 2017) have emerged as a prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks.

Original: By averaging predictions or by voting, DE achieve high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.
Revised: By averaging predictions or by voting, DE achieves high accuracy and robustness since potentially unreliable predictions are exposed via the disagreement between individuals.

Original: Thanks to the simplicity and effec- tiveness of the ensembling strategy (Dietterich, 2000), DE have become largely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: Thanks to the simplicity and effectiveness of the ensembling strategy (Dietterich, 2000), DE have become widely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: DE tick all boxes in therequirements list for real-world applications, exceptfor computational efficiency.
Revised: DE meet most of the real-world application requirements except computational efficiency.

Original: In fact, DE arein practice computationally demanding (required memory storage, number of operations,  inference time) bothfor training and testing as their costs grow linearly with the size of the ensembles.
Revised: Specifically, DE are computationally demanding in terms of memory storage, number of operations, and inference time during both training and testing, as their costs grow linearly with the number of individuals.

Original: Their computational costs are therefore prohibitive under tight hardware constraints.
Revised: Their computational costs are, therefore, prohibitive under tight hardware constraints.

Original: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020),architectures with ensemble-imitating layers (Wen et al., 2019; Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016; Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018),  efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).
Revised: This limitation of DE has inspired numerous approaches proposing computationally efficient alternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020), ensemble-imitating layers (Wen et al., 2019; Havasi et al., 2020; Ram´e et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016; Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from a single training run (Huang et al., 2017; Garipov et al., 2018), and efficient Bayesian Neural Networks (Maddox et al., 2019; Franchi et al., 2020).

Original: These approaches typically improve storage usage, train cost or inference time at the cost of lower accuracy andlower diversity in the predictions.
Revised: These approaches typically improve storage usage, train cost, or inference time at the cost of lower accuracy and diversity in the predictions.

Original: An essential property of ensembles that enables improved predictive uncertainty estimation is related to the diversity of predictions in the ensemble.
Revised: An essential property of ensembles to improve predictive uncertainty estimation is related to the diversity of its predictions.

Original: Perrone & Cooper (1992) have shown that the independence of individuals is critical to the success of ensembling.
Revised: Perrone & Cooper (1992) show that the independence of individuals is critical to the success of ensembling.

Original: Fort et al.
Revised: Fort et al.

Original: (2019) argue that the diversity of DE due to randomness from weight initialization, data augmentation and batching,  stochastic gradient updates,etc., is superior tovarious other efficient ensembling approaches, in spite of their predictive performance boosts.
Revised: (2019) argue that the diversity of DE, due to randomness from weight initialization, data augmentation and batching, and stochastic gradient updates, is superior to other efficient ensembling alternatives, despite their predictive performance boosts.

Original: Few works manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes,  image throughput, etc.).
Revised: Few approaches manage to mirror this property of DE in a computationally efficient manner close to a single DNN (in terms of memory usage, number of forward passes, and image throughput).

Original: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular having a set of independent networks, in a computationally efficient manner.
Revised: In this work, we aim to design a DNN architecture that closely mimics properties of ensembles, in particular, having a set of independent networks, in a computationally efficient manner.

Original: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.
Revised: Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to a single large model.

Original: We build upon this idea and devise a strategy based on small networks towards matching the performance of an ensemble of large networks.
Revised: We build upon this idea and devise a strategy based on small networks trying to match the performance of an ensemble of large networks.

Original: To this end we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.
Revised: To this end, we leverage grouped convolutions (Krizhevsky et al., 2012) to delineate multiple subnetworks within the same network.

Original: The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models.
Revised: The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models.

Original: This  enables fast training and inference times, while predictive uncertainty quantification is close to DE (Figure 1).
Revised: This method enables fast training and inference times while predictive uncertainty quantification is close to DE (Figure 1).

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured-sparsity for Deep Ensembles; • We extensively evaluate PE in terms of accuracy, calibration, OOD detection, and dataset shift on classification and regression tasks.
Revised: • We propose Packed-Ensembles (PE), an efficient ensembling architecture relying on grouped convolutions, as a formalization of structured sparsity for Deep Ensembles; • We extensively evaluate PE regarding accuracy, calibration, OOD detection, and distribution shift on classification and regression tasks.

Original: We demonstrate that PE achieves state-of-theart predictive uncertainty quantification.
Revised: We show that PE achieves state-of-the-art predictive uncertainty quantification.

Original: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.
Revised: • We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we present the formalism for this work and offer a short background on grouped convolutions and ensembles of DNNs.
Revised: In this section, we present the formalism for this work and offer a brief background on grouped convolutions and ensembles of DNNs.

Revised: Appendix A summarizes the main notations in Table 3.

Original: 2. B ACKGROUND ON CONVOLUTIONS
Revised: 2. B ACKGROUND ON CONVOLUTIONS

Original: The convolutional layer (LeCun et al., 1989) consists in a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C 
Revised: The convolutional layer (LeCun et al., 1989) consists of a series of cross-correlations between feature maps h j ∈ R C j × H j × W j regrouped in batches of size B and a weight tensor ω j ∈ R C j +1 × C

Original: j+1 × C j × s 2 j  with C j , H j , W j three integers that represent respectively the number of channels, the height and the width of h j .
Revised: j × s 2 j [Figure] with C j , H j , W j three integers representing the number of channels, the height and the width of h j respectively.

Original: C j +

Revised: C j +

Original: and s j are also two integers correspondingrespectively to the number of channels of h j +1 (the output of the layer) and the kernel size.
Revised: and s j are also two integers corresponding to the number of channels of h j + (the output of the layer) and the kernel size.

Original: Finally, j is the layer’s index and will be fixed in the following formulae.
Revised: Finally, j is the layer’s index and will be fixed in the following formulae.

Original: The main notations are detailed in Table 3, in the supplementary

Original: [Figure]

Original: material.

Original: The bias of convolution layers will be omitted in the following for simplicity.
Revised: The bias of convolution layers will be omitted in the following for simplicity.

Original: Hence the output value of the convolution layer, denoted ⊛ , is:
Revised: Hence the output value of the convolution layer, denoted ⊛ , is:

Original: [Equation]
Revised: [Equation]

Original: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ
Revised: where c ∈ (cid:74) 0 , C j +1 − 1 (cid:75) is the index of the considered channel of the output feature map, ⋆ is the classical 2D cross-correlation operator, and z j is the pre-activation feature map such that h j = ϕ

Original: ( z j ) with ϕ an activation function.
Revised: ( z j ) with ϕ an activation function.

Original: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in
Revised: To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in

Original: ResNext (Xie et al., 2017) to train several DNN branches in parallel.
Revised: ResNeXt (Xie et al., 2017) to train several DNN branches in parallel.

Original: The grouped convolution operation with γ groups and weights ω iγ ∈ R C
Revised: The grouped convolution operation with γ groups and weights ω iγ ∈ R C

Original: j +1 ×
Revised: j +1 ×

Original: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.
Revised: Cjγ × s 2 j is given in (2), γ dividing C j for all layers.

Original: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC [Equation] (cid:107) , which only uses 1 γ of the input channels.
Revised: Any output channel c is produced by a specific group (set of filters), identified by the integer (cid:106) γcC j + (cid:107) , which only uses 1 γ of the input channels:

Original: [Equation]
Revised: [Equation]

Original: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×
Revised: The grouped convolution layer is mathematically equivalent to a classical convolution where the weights are multiplied element-wise by the binary tensor mask m ∈ { 0 , 1 } C j +1 ×

Original: C j × s 2 j such that mask jm ( k, l, : , :)
Revised: C j × s 2 j such that mask jm ( k, l, : , :)

Original: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)
Revised: = 1 if (cid:106) γlC j (cid:107) = (cid:106) γkC j +1 (cid:107)

Original: =
Revised: =

Original: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .
Revised: m for each group m ∈ (cid:74) 0 , γ − 1 (cid:75) .

Original: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) m = 0 mask j m and the grouped convolution can therefore be rewritten as:
Revised: The complete layer mask is finally defined as mask j = γ − 1 (cid:80) m =0 mask jm and the grouped convolution can therefore be rewritten as z j +1 = h

Original: [Equation] where ◦ is the Hadamard product.
Revised: j ⊛ (cid:0) ω j ◦ mask j (cid:1) , where ◦ is the Hadamard product.

Original: 2. B ACKGROUND ON D EEP E NSEMBLES
Revised: 2. B ACKGROUND ON D EEP E NSEMBLES

Original: For a classification problem, let us define a dataset D
Revised: For a classification problem, let us define a dataset D

Original: = { x i , y i } |D| i =1 containing |D| pairs of samples x i ∈
Revised: = { x i , y i } |D| i =1 containing |D| pairs of samples x i ∈

Original: R H j × W j and one-hot-encoded labels
Revised: R H j × W j and one-hot-encoded labels

Original: y i ∈
Revised: y i ∈

Original: R N C modeled as the realisation of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.
Revised: R N C modeled as the realisation of a joint distribution P ( X,Y ) where N C is the number of classes in the dataset.

Original: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i
Revised: The input data x i is processed via a neural network f θ which is a parametric probabilistic model such that ˆ y i

Original: =
Revised: =

Original: f θ ( x i ) =
Revised: f θ ( x i ) =

Original: P ( Y = y
Revised: P ( Y = y

Original: i
Revised: i

Original: | X = x i ; θ ) .
Revised: | X = x i ; θ ) .

Original: This approach consists in considering the prediction ˆ y as parameters of a Multinoulli distribution.
Revised: This approach consists in considering the prediction ˆ y as parameters of a Multinoulli distribution.

Original: To improve the accuracy of predictions but also the accuracy of the predicted uncertainties and the detection of OOD samples, Lakshminarayanan et al.

Original: (2017) have proposed to ensemble M randomly equivalent architectures 3,3x3,
Revised: equivalent architectures 3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =
Revised: Pool 12,3x3, 48,1x1, 24,1x1, xxxxxx3,3x3,36groups =

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Original: 72,1x1,144groups =
Revised: 72,1x1,144groups =

Original: Rearrange xxx3,3x3,
Revised: Rearrange xxx3,3x3,

Original: Av.
Revised: Av.

Original: Pool 36,3x3,72groups
Revised: Pool 36,3x3,72groups

Original: =
Revised: =

Original: 144,1x1,30groups =
Revised: 144,1x1,30groups =

Revised: 72,1x1,144groups =

Revised: [Figure]

Revised: To improve the quality of both predictions and estimated uncertainties, as well as the detection of OOD samples, Lakshminarayanan et al.

Original: 72,1x1,144groups =a) b) c) initialized deep Neural Networks as a large predictor called Deep Ensembles.
Revised: (2017) propose to ensemble M randomly initialized DNNs as a large predictor called Deep Ensembles.

Original: These ensembles can be seen as a discrete approximation of the intractable bayesian marginalization on the weights, according to Wilson & Izmailov (2020).
Revised: These ensembles can be seen as a discrete approximation of the intractable Bayesian marginalization on the weights, according to Wilson & Izmailov (2020).

Original: If we note { θ m } M − 1 m =

Revised: If we note { θ m } M − 1 m =

Original: the set of trained weights for the M DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (4).
Revised: the set of trained weights for the M DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (3).

Original: [Equation]
Revised: [Equation]

Original:  P ACKED -E NSEMBLES
Revised:  P ACKED -E NSEMBLES

Original: In this section, we present how toefficiently train multiple subnetworks using grouped convolution.
Revised: This section describes how to train multiple subnetworks using grouped convolution efficiently.

Original: Then, we explain how our new architectures are equivalent to training several networks in parallel.
Revised: Then, we explain how our new architectures are equivalent to training several networks in parallel.

Original: 3. R EVISITING D EEP E NSEMBLES
Revised: 3. R EVISITING D EEP E NSEMBLES

Original: Although Deep Ensembles provide undisputed benefits, they also come with the major drawback that the training time and the memory usage in inference increase linearly with the number of networks.
Revised: Although Deep Ensembles provide undisputed benefits, they also come with the significant drawback that the training time and the memory usage in inference increase linearly with the number of networks.

Original: To alleviate these problems, we propose assembling small subnetworks, which are  DNNs with fewer parameters.
Revised: To alleviate these problems, we propose assembling small subnetworks, which are essentially DNNs with fewer parameters.

Original: Moreover, while ensembles to this day have  been trained sequentially, we propose to leverage grouped convolutions toaccelerate massively  their training and inference computations thanks to their smaller size.
Revised: Moreover, while ensembles to this day have mostly been trained sequentially, we suggest leveraging grouped convolutions to massively accelerate their training and inference computations thanks to their smaller size.

Original: The propagation of grouped convolutions with M groups, M being the number of subnetworks in the ensemble, ensures that the latters are trained independently while dividing their encoding dimension by a factor M .
Revised: The propagation of grouped convolutions with M groups, M being the number of subnetworks in the ensemble, ensures that the subnetworks are trained independently while dividing their encoding dimension by a factor M .

Original: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.
Revised: More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.

Original: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate  the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as a expansion factor.
Revised: To create Packed-Ensembles (illustrated in Figure 2), we build on small subnetworks but compensate for the dramatic decrease of the model capacity by multiplying the width by the hyperparameter α , which can be seen as an expansion factor.

Original: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.
Revised: Hence, we propose Packed-Ensembles- ( α, M, 1) as a flexible formalization of ensembles of small subnetworks.

Original: For an ensemble of M subnetworks,
Revised: For an ensemble of M subnetworks,

Original: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor αM and the inference of our ensemble is computed with the following formula:
Revised: Packed-Ensembles- ( α, M, 1) therefore modifies the encoding dimension by a factor α M and the inference of our ensemble is computed with the following formula:

Original: [Equation]
Revised: [Equation]

Original: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×
Revised: where ω j,α is the weight of the layer j of dimension ( αC j +1 ) × ( αC j ) ×

Original: s 2 j .
Revised: s 2 j .

Original:  In the following, we also add another hyperparameter γ which corresponds to the number of groups of each subnetworks of the Packed-Ensembles and creates another level of sparsity to our ensembles.
Revised: [Figure] m= m= [Figure] In the following, we introduce another hyperparameter γ corresponding to the number of groups of each subnetwork of the Packed-Ensembles, creating another level of sparsity.

Original: These groups are also called ”subgroups” and are applied to the different subnetworks.
Revised: These groups are also called ”subgroups” and are applied to the different subnetworks.

Original: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , where the hyperparametersare in the parentheses.
Revised: Formally, we denote our technique Packed-Ensembles- ( α, M, γ ) , with the hyperparameters in the parentheses.

Original: In this work, we considerthe case of a constant number of subgroups across the layers and therefore γ divides αC j for all j .m= [Figure] [Figure]
Revised: In this work, we consider a constant number of subgroups across the layers; therefore, γ divides αC j for all j .

Original: m= m=

Original: [Figure]

Original: 3. C OMPUTATIONAL COST
Revised: 3. C OMPUTATIONAL COST

Original: In a convolutional setting, the number of parametersin a layer involving C j input channels, C j + output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)
Revised: For a convolutional layer, the number of parameters involving C j input channels, C j +1 output channels, kernels of size s j and γ subgroups is equal to M × (cid:104)

Original: αC j M αC j +1 M s 2 j γ − 1 (cid:105) .
Revised: [Equation]

Original: The same formula applies to dense layers as 1 × 1 convolutions.
Revised: The same formula applies to dense layers as 1 × 1 convolutions.

Original: Whenever the architectures of the subnetworks are fully convolutional or dense, two specific cases emerge.
Revised: Two cases emerge when the architectures of the subnetworks are fully convolutional or dense.

Original: If α = √ M , the number of parameters in the ensemble is equal to the number of parameters in a single model.
Revised: If α = √ M , the number of parameters in the ensemble equals the number of parameters in a single model.

Original: With α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent in size to Deep Ensembles).
Revised: With α = M , each subnetwork corresponds to a single model (and their ensemble is therefore equivalent in size to DE).

Original: 3. I MPLEMENTATION DETAILS
Revised: 3. I MPLEMENTATION DETAILS

Original: In this paper, we propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.
Revised: We propose a simple way of designing efficient ensemble convolutional layers using grouped convolutions.

Original: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) on Figure 3, by the parallel implementations (b) and (c) .
Revised: To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) in Figure 3, with the parallel implementations (b) and (c) .

Original: Figure 3 summarizes different equivalent architectures for a simple ensemble of 
Revised: Figure 3 summarizes different equivalent architectures for a simple ensemble of M =

Original: M = 3 DNNs with 3 convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .
Revised: DNNs with three convolutional layers and a final dense layer (equivalent to a 1 × 1 convolution) with α = γ = 1 .

Original: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the ”rearrange”operation).
Revised: In (b) , we propose to stack the feature maps on the channel dimension (denoted as the rearrange operation).

Original: This yields a feature map h j , of size M × C
Revised: This yields a feature map h j , of size M × C

Original: i × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.
Revised: i × H j × W j regrouped by batches of size only BM , with B the batch size of the ensemble.

Original: One solution to keep the same batch size is to repeat the batch M times so that its size is equal to B after the rearrangement.
Revised: One solution to keep the same batch size is to repeat the batch M times so that its size equals B after the rearrangement.

Original: Using convolutions with M groups and γ subgroups per subnetworks, each feature map is convoluted separately by each subnetwork and yields its own independent output.
Revised: Using convolutions with M groups and γ subgroups per subnetwork, each feature map is convoluted separately by each subnetwork and yields its own independent output.

Original: Grouped convolutions are propagated until the end to ensure that gradients stay independent between subnetworks.
Revised: Grouped convolutions are propagated until the end to ensure that gradients stay independent between subnetworks.

Original: Other operations such as 
Revised: Other operations, such as Batch

Original: Batch Normalization (Ioffe & Szegedy, 2015) can be applied directly as long as they can be grouped or have independent actions on each channel.
Revised: Normalization (Ioffe & Szegedy, 2015), can be applied directly as long as they can be grouped or have independent actions on each channel.

Original: Figure 4a illustrates the mask used to code PackedEnsembles in the case where M = 2 .
Revised: Figure 4a illustrates the mask used to code PackedEnsembles in the case where M = 2 .

Original: Similarly, Figure 4b shows the mask with M = 2 and γ = 2 .
Revised: Similarly, Figure 4b shows the mask with M = 2 and γ = 2 .

Original: Finally, (b) and (c) are also equivalent.
Revised: Finally, (b) and (c) are also equivalent.

Original: It is indeed possible to replace the ”rearrange” operation and the first grouped convolution by a standard convolution if the same images are to be provided at the same time to all the subnetworks.
Revised: It is indeed possible to replace the rearrange operation and the first grouped convolution with a standard convolution if the same images are to be provided simultaneously to all the subnetworks.

Original: We confirm in Appendix F that this procedure is not detrimental to the performance of the ensemble and we take advantage from this property to provide this final optimization and simplification.
Revised: We confirm in Appendix F that this procedure is not detrimental to the ensemble’s performance, and we take advantage of this property to provide this final optimization and simplification.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To validate the performance of our method, we conduct experiments on classification tasks andwe measure the effect of the parameters α and γ .
Revised: To validate the performance of our method, we conduct experiments on classification tasks and measure the influence of the parameters α and γ .

Original: Regression tasks are detailed in Appendix N.
Revised: Regression tasks are detailed in Appendix N.

Original: 4. D ATASETS AND ARCHITECTURES
Revised: 4. D ATASETS AND ARCHITECTURES

Original: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to  differentdata complexities.
Revised: First, we demonstrate the efficiency of Packed-Ensembles on CIFAR-10 and CIFAR100 (Krizhevsky, 2009), showing how the method adapts to tasks of different complexities.

Original: As we propose to replace a single model architecture with several subnetworks, we study the behavior of Packed-Ensembles onarchitectures with various sizes: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet28-10 (Zagoruyko & Komodakis, 2016).
Revised: As we propose to replace a single model architecture with several subnetworks, we study the behavior of PE on various sizes architectures: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet2810 (Zagoruyko & Komodakis, 2016).

Original: We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and three other approximated ensembles from the literature: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020) and Masksembles (Durasov et al., 2021).
Revised: We compare it against Deep Ensembles (Lakshminarayanan et al., 2017) and three other approximated ensembles from the literature: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020), and Masksembles (Durasov et al., 2021).

Original: Secondly, we report our results for Packed-Ensembles on ImageNet, which we compare against all baselines.
Revised: Second, we report our results for Packed-Ensembles on ImageNet (Deng et al., 2009), which we compare against all baselines.

Original: We run experiments with ResNet-50 and ResNet-50x4.
Revised: We run experiments with ResNet-50 and ResNet-50x4.

Original: All training runs are started from scratch.
Revised: All training runs are started from scratch.

Original: 4.1. M ETRICS 
Revised: 4.1. M ETRICS , OOD DATASETS , AND IMPLEMENTATION

Original: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and  Negative Log-Likelihood (NLL).
Revised: We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and the Negative Log-Likelihood (NLL).

Original: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2, and measure the quality of the OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in % similarly to Hendrycks & Gimpel (2017)
Revised: We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties 2 and measure the quality of the OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in %, similarly to Hendrycks & Gimpel (2017).

Original: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).
Revised: We use accuracy as the validation criterion (i.e., the final trained model is the one with the highest accuracy).

Original: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble.
Revised: During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble.

Original: We define the prediction confidence asthe value of this maximum  (also called maximum softmax probability).
Revised: We define the prediction confidence as this maximum value (also called maximum softmax probability).

Original: For the out-of-distribution detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution(ID) data and OOD data using the maximum softmax probability asOOD criterion.
Revised: For OOD detection tasks on CIFAR-10 and CIFAR-100, we use the SVHN dataset (Netzer et al., 2011) as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution and OOD data using the maximum softmax probability as criterion.

Original: We discuss the different OOD criteria in appendix E. 
Revised: We discuss the different OOD criteria in appendix E. For ImageNet, we use two out-ofdistribution datasets:

Original: For ImageNet, we use two out-of-distribution datasets: ImageNet-O(IO) (Hendrycks et al., 2021b) and Texture(T) (Wang et al., 2022), and use the Mutual Information ( MI ) as  criterion for the ensembles techniques (see Appendix E for details on MI ) and the maximum softmax probability for the single model and MIMO.
Revised: ImageNet-O (Hendrycks et al., 2021b) and Texture (Wang et al., 2022), and use the Mutual Information (MI) as a criterion for the ensembles techniques (see Appendix E for details on MI) and the maximum softmax probability for the single model and MIMO.

Original: To measure the robustness under distribution shift, we use ImageNetR (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denotedrespectively rAcc, rECE, and rNLL on this dataset.
Revised: To measure the robustness under distribution shift, we use ImageNet-R (Hendrycks et al., 2021a) and evaluate the Accuracy, ECE, and NLL, denoted rAcc, rECE, and rNLL on this dataset, respectively.

Original: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.
Revised: We implement our models using the PyTorch-Lightning framework built on top of PyTorch.

Original: Both are open-source Python frameworks.
Revised: Both are open-source Python frameworks.

Original: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.
Revised: Appendix B and Table 4 detail the hyper-parameters used in our experiments across architectures and datasets.

Original: Most trainings are completed on a singleNvidia
Revised: Most training instances are completed on a single

Original:  RTX 3090 except on ImageNet for which we use 2 to 8 Nvidia A100-80GB.
Revised: Nvidia RTX 3090 except for ImageNet, for which we use 2 to 8 Nvidia A100-80GB.

Original: 4.1. R ESULTS
Revised: 4.1. R ESULTS

Original: Table 1 presents the average performance over five runs on the classification task  using the hyperparameters on Table 4.
Revised: Table 1 presents the average performance for the classification task over five runs using the hyperparameters in Table 4.

Original: We showcase that  in theparticular setting of α = 2 and γ = 2 ,PackedEnsembles yields similar results to Deep Ensembles while having a lower memory cost than the single model.
Revised: We demonstrate that Packed-Ensembles, in the setting of α = 2 and γ = 2 , yields similar results to Deep Ensembles while having a lower memory cost than a single model.

Original: On CIFAR-10 (Krizhevsky, 2009), we notice that its relative performance  compared to Deep Ensembles seems to increase as the original architecture gets bigger.
Revised: For CIFAR-10, the relative performance of PE compared to DE appears to increase as the original architecture becomes larger.

Original: For ResNet-18, the method matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.
Revised: When using ResNet-18, Packed-Ensembles matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others.

Revised: However, using

Original: Using ResNet-50, both models seem to perform equivalently, while PackedEnsembles slightly outperforms Deep Ensembles interms of classification performance with Wide
Revised: ResNet-50, both models seem to perform similarly, and PE slightly outperforms DE in classification performance with WideResNet [Table] [Table] 5.19 MIMO ( ρ = 1 )

Revised: C

Original: ResNet28-10.
Revised: R 95. 1.30 Masksembles C [Table]

Revised: 5.19 Packed-Ensembles

Revised: C

Revised: [Table]

Revised: [Table]

Revised: 1.00 Deep Ensembles

Revised: C

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Single Model

Revised: [Table]

Revised: WR

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 95 BatchEnsemble

Revised: C

Revised: WR 95.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 23.81 MIMO

Revised: [Table]

Revised: C

Revised: WR 94.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 96 Masksembles

Revised: [Table]

Revised: WR

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: On CIFAR-100, Packed-Ensembles is outperformed by Deep Ensembles on ResNet-18.
Revised: [Table] 23.82 Packed-Ensembles [Table] WR 96.

Revised: [Table]

Revised: 4.06 Deep Ensembles

Revised: C

Revised: WR 95.

Revised: Single Model

Revised: [Table]

Original: Yet, we argue that ResNet-18 architecture does not have enough representation capacity to be divided into
Revised: [Table] [Table] [Table] .56 BatchEnsemble

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  subnetworks for CIFAR-100.
Revised: [Table] [Table] On CIFAR-100, Deep Ensembles outperforms Packed-Ensembles on ResNet-18. However, we argue that ResNet-18 architecture need more representation capacity to be divided into subnetworks for CIFAR-100.

Original: Indeed, when we look at the results of ResNet-50, we can see that Packed-Ensembles has better results than Deep Ensembles.
Revised: Indeed, when we look at the results of ResNet-50, we can see that PackedEnsembles has better results than Deep Ensembles.

Original: This  demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.
Revised: This analysis demonstrates that, given a sufficiently large network, Packed-Ensembles is able to match Deep Ensembles with only 16% of its parameters.

Original: In appendix D, we discuss the influence of the representation capacity.
Revised: In Appendix D, we discuss the influence of the representation capacity.

Original: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.
Revised: Based on the results in Table 2, we can conclude that Packed-Ensembles improves uncertainty quantification for OOD and distribution shift on ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.

Original: 4.1. S TUDY ON THE PARAMETERS α AND γ
Revised: 4.1. S TUDY ON THE PARAMETERS α AND γ

Original: Table 1 reports resultsonly for α = 2 and γ = 2 , however depending on the task, the architecture used and the available memory, one might want to tune those parameters to fit more precisely its needs.
Revised: Table 1 reports results for α = 2 and γ = 2 . However, the optimal values of these hyperparameters depend on the balance between computational cost and performance.

Original:  Figures 6 and 7 showcase the performance evolution of the Packed-Ensembles as the α values varies.
Revised: To help users strike the best compromise, we propose Figures 6 and 7 in Appendix D, which illustrate the impact of changing α on the performance of Packed-Ensembles.

Original:  D ISCUSSIONS
Revised:  D ISCUSSIONS

Original: We have shown that Packed-Ensembles has interesting properties, especially by providing a similar quality of Uncertainty Quantification as Deep Ensemble while using a reduced architecture and  computing cost.
Revised: We have shown that Packed-Ensembles has attractive properties, mainly by providing a similar quality of uncertainty quantification as Deep Ensembles while using a reduced architecture and [Table] computing cost.

Original: Several questions can be raised and we conducted some studies, detailed in the
Revised: Several questions can be raised, and we conducted some studies - detailed in the

Original: Appendix section, in order to bring possible answers.
Revised: Appendix sections - to provide possible answers.

Original: [Table] Discussion on the sparsity As described in section 3, one could interpret Packed-Ensembles as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.
Revised: Discussion on the sparsity As described in section 3, one could interpret PE as leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components.

Original: In Appendix C, by using a simplified model, we are able to bound  the approximation error based on the Kullback-Leibler divergence between the Deep Ensemble and its pruned version.
Revised: In Appendix C, by using a simplified model, we propose a bound of the approximation error based on the Kullback-Leibler divergence between the DE and its pruned version.

Original: This bound depends on the density of ones in the mask p , and more specifically, depends onthe terms ( p  −1) , p ( p − 1) and ( p − 1) 2 /p2 .
Revised: This bound depends on the density of ones in the mask p , and, more specifically, depends on p (1 − p ) and (1 − p ) 2 /p .

Original: By manipulating these terms, corresponding to modifying the number of subnetworks M , the number of groups γ  and the dilatation faction α , we could theoreticallybe able to control the approximation error.
Revised: By manipulating these terms, corresponding to modifying the number of subnetworks M , the number of groups γ , and the dilation factor α , we could theoretically control the approximation error.

Original: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two main sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.
Revised: On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two primary sources of stochasticity: the random initialization of the model’s parameters and the shuffling of the batches.

Original: A last source of stochasticity is introduced duringthe trainingphase by the non-deterministic behaviour of the backward algorithms.
Revised: A last source of stochasticity is introduced during training by the non-deterministic behavior of the backpropagation algorithms.

Original: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.
Revised: In Appendix F, we study the function space diversities which arise from every possible combination of these sources.

Original: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seem to emerge to predict the best combination.
Revised: It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seems to emerge to predict the best combination.

Original: Especially, we highlight that even the only use of nondeterministic algorithms introduces enough diversity between each subnetwork of the ensemble.
Revised: Specifically, we highlight that even the only use of non-deterministic algorithms introduces enough diversity between each subnetwork of the ensemble.

Original: Ablation study We perform ablations studiesin order to assess the impact of the parameters M , γ , and α on the performances of Packed-Ensembles.
Revised: Ablation study We perform ablation studies to assess the impact of the parameters M , α , and γ on the performance of Packed-Ensembles.

Original: The details of this study are given in Appendix
Revised: Appendix D provides in-depth details of this study.

Original: D. No explicit behavior appears from the results we obtained.
Revised: No explicit behavior appears from the results we obtained.

Original: A trend seems to show that a higher number of subnetworks is helpful in obtaining a better OOD detection, but the improvement interms of AUPR is not significant.
Revised: A trend shows that a higher number of subnetworks helps get better OOD detection, but the improvement in AUPR is not significant.

Original: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.
Revised: Training speed Depending on the chosen hyperparameters α , M , and γ , PE may have fewer parameters than the single model, as shown in Table 1.

Original: This translates into an expected lower number of operations.
Revised: This translates into an expected lower number of operations.

Original: However, a study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.
Revised: A study of the training and inference speeds, developed in Appendix H, shows that using PE-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances.

Original: Yet, this also hints that the actual speedup is not optimal despite the important acceleration offered by 16 bits floating points.
Revised: However, this also hints that the group-convolution speedup is not optimal despite the significant acceleration offered by 16-bits floating points.

Original: OOD criteria The  criterion for discriminating OOD elements is based on the maximum softmax probability.
Revised: OOD criteria The maximum softmax probability is often used as criterion for discriminating OOD elements.

Original: However, this criterion is not unique, and others can be used, such as the Mutual Information, the maximum logit, or the Shannon entropy of the mean prediction.
Revised: However, this criterion is not unique, and others can be used, such as the Mutual Information, the maximum logit, or the Shannon entropy of the mean prediction.

Original: Although no relationship is expected between this criterion and the method of Packed-Ensembles, we obtained different performances in OOD detection according to the selected criterion.
Revised: Although no relationship is expected between this criterion and PE, we obtained different performances in OOD detection according to the selected criterion.

Original: The results  are detailed in Appendix E, andthey show that an approach based on the maximum logit seems to give the best results in detecting OOD.
Revised: The results on CIFAR-100 are detailed in Appendix E and show that an approach based on the maximum logit seems to give the best results in detecting OOD.

Original: It should be noted thatthis discussion focuses primarily on CIFAR-100 and that the notion of OOD depends on the training distribution.
Revised: It should be noted that the notion of OOD depends on the training distribution.

Original: Such a discussion does not necessarily generalize to all datasets.
Revised: Such a discussion does not necessarily generalize to all datasets.

Original: Indeed, preliminary results have shown that Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.
Revised: Indeed, preliminary results have shown that Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.

Original:  R ELATED W ORK
Revised:  R ELATED WORK

Original: Ensembles and uncertainty quantification.
Revised: Ensembles and uncertainty quantification.

Original: Bayesian Neural Networks (BNNs) (MacKay, 1992;
Revised: Bayesian Neural Networks (BNNs) (MacKay, 1992;

Original: Neal, 1995) are the cornerstone and main source of inspiration for uncertainty quantification in deep learning.
Revised: Neal, 1995) are the cornerstone and primary source of inspiration for uncertainty quantification in deep learning.

Original: In spite of the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain difficult to scale and train for large DNN architectures (Dusenberry et al., 2020).
Revised: Despite the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain challenging to scale and train for large DNN architectures (Dusenberry et al., 2020).

Original: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of
Revised: DE (Lakshminarayanan et al., 2017) arise as a practical and efficient instance of

Original: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).
Revised: BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson & Izmailov, 2020).

Original: DE are currently the best performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).
Revised: DE are currently the best-performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).

Original: Efficient ensembles.
Revised: Efficient ensembles.

Original: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost have inspired a large cohort of approaches aiming to mitigate it.
Revised: The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost, have inspired a large cohort of approaches aiming to mitigate it.

Original: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork specific parameters trained in parallel.
Revised: BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks to an efficient parameterization of subnetwork-specific parameters trained in parallel.

Original: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.
Revised: MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using a multi-input multi-output configuration.

Original: A single network can be used in ensemble-mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).
Revised: A single network can be used in ensemble mode by disabling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021).

Original: Liu et al.
Revised: Liu et al.

Original: (2022) leverage the sparse networks training algorithm of Mocanu et al.
Revised: (2022) leverage the sparse networks training algorithm of Mocanu et al.

Original: (2018) to produce ensembles of sparse networks.
Revised: (2018) to produce ensembles of sparse networks.

Original: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020),  by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).
Revised: Ensembles can be computed from a single training run by collecting intermediate model checkpoints (Huang et al., 2017; Garipov et al., 2018), by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019; Franchi et al., 2020), and by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020).

Original: However most of these approaches require multiple forward passes.
Revised: However, most of these approaches require multiple forward passes.

Original: Neural network compression.
Revised: Neural [Table]

Original: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by-design, relying on e.g., channel shuffling (Zhang & Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.
Revised: The most intuitive approach for reducing the size of a model is to employ DNNs that are memory-efficient by design, relying on, e.g., channel shuffling (Zhang & Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them.

Original: Some of the most popular architectures the leverage such models are: SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b),  MobileNet-v3 (Howard et al., 2019).
Revised: Some of the most popular architectures that leverage such models are SqueezeNet (Iandola et al., 2016), ShuffleNet (Zhang et al., 2018b), and MobileNet-v3 (Howard et al., 2019).

Original: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;
Revised: Some approaches conduct automatic model size reduction, e.g., network sparsification (Molchanov et al., 2017;

Original: Louizos et al., 2018; Frankle & Carbin, 2018;
Revised: Louizos et al., 2018; Frankle & Carbin, 2018;

Original: Tartaglione et al., 2022).
Revised: Tartaglione et al., 2022).

Original: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency.
Revised: These approaches aim at removing as many parameters as possible from the model to improve memory and computation efficiency, also at train time Bragagnolo et al.

Original: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.
Revised: Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.

Original: Grouped convolutions.
Revised: Grouped convolutions.

Original: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.
Revised: To the best of our knowledge, grouped convolutions (group of convolutions) were introduced by Krizhevsky et al.

Original: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.
Revised: Enabling the computation of several independent convolutions in parallel, they developed the idea of running a single model on multiple GPU devices.

Original: Xie et al.
Revised: Xie et al.

Original: (2017) demonstrate that using grouped convolutions lead to accuracy improvements and model complexity reduction.
Revised: (2017) demonstrate that using grouped convolutions leads to accuracy improvements and model complexity reduction.

Original: So far, grouped convolutions have been used primarily for computational efficiency but also to compute multiple output branches in parallel (Chen & Shrivastava, 2020).
Revised: So far, grouped convolutions have been used primarily for computational efficiency but also to compute multiple output branches in parallel (Chen & Shrivastava, 2020).

Original: Here, we re-purpose them for delineating multiple subnetworks within a network andfor efficiently training an ensemble of such subnetworks.
Revised: PE re-purpose them to delineate multiple subnetworks within a network and efficiently train an ensemble of such subnetworks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a new ensemble framework: Packed-Ensembles, which is able to approximate Deep Ensembles in terms of uncertainty quantification and accuracy.
Revised: We propose a new ensemble framework, Packed-Ensembles, that can approximate Deep Ensembles in terms of uncertainty quantification and accuracy.

Original: Our work provides several new findings: first, we show thatensembling independent small  neural networks can be equivalent to ensembling independent deep neural networks.
Revised: Our research provides several new findings. First, we show that small independent neural networks can be as effective as large, deep neural networks when used in ensembles.

Original: Secondly, we demonstrate that not all sources of diversity are needed to improve the diversity of the ensemble.
Revised: Secondly, we demonstrate that not all sources of diversity are essential for improving ensemble diversity.

Original: Thirdly, we show that Packed-Ensembles is more stable thana single DNN.
Revised: Thirdly, we show that Packed-Ensembles are more stable than single DNNs.

Original: Fourthly, we highlight that there is a trade-off betweenthe accuracy and the size ofthe parameters, and Packed-Ensembles allows us to have flexible and efficient ensembling.
Revised: Fourthly, we highlight that there is a trade-off between accuracy and the number of parameters, and Packed-Ensembles enables us to create flexible and efficient ensembles.

Original: In the future, we intend to explore Packed-Ensembles for more complex  tasks.
Revised: In the future, we intend to explore Packed-Ensembles for more complex downstream tasks.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: Alongside this paper, we provide the source code of Packed-Ensembles layers.
Revised: Alongside this paper, we provide the source code of Packed-Ensembles layers.

Original: Moreover, two notebooks enable the training of Packed-Ensembles based on the ResNet-50 architecture, both on CIFAR-10 and CIFAR-100 (public datasets).
Revised: Additionally, we have created two notebooks demonstrating how to train ResNet-50-based Packed-Ensembles using public datasets such as CIFAR-10 and CIFAR-100.

Original: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.
Revised: To ensure reproducibility, we report the performance given a specific random seed with a deterministic training process.

Original: Furthermore, it should be noted that the source code contains two PyTorch Module classes toefficiently produce Packed-Ensembles.
Revised: Furthermore, it should be noted that the source code contains two PyTorch Module classes to produce Packed-Ensembles efficiently.

Original: The idea would indeed be to release a Python package to provide easier access to Packed-Ensembles layers.

Original: A readme file at the root of the project details how to install and run experiments.
Revised: A readme file at the root of the project details how to install and run experiments.

Original: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).
Revised: In addition, we showcase how to get Packed-Ensembles from LeNet (LeCun et al., 1998).

Revised: To further promote accessibility, we have created an open-source pip-installable PyTorch package, torch-uncertainty, that includes Packed-Ensembles layers.

Revised: With these resources, we hope to encourage the broader research community to engage with and build upon our work.

Original:  E THICS
Revised:  E THICS

Original: The purpose of this paper is to provide a  method fora better estimation of Deep Learning model uncertainty.
Revised: The purpose of this paper is to propose a new method for better estimations of uncertainty for deep-learning-based models.

Original: Nevertheless, we acknowledge their limitation, which could become critical when applied to safety systems for instance.
Revised: Nevertheless, we acknowledge their limitations, which could become particularly concerning when applied to safety-critical systems.

Original: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.
Revised: While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems.

Original: We show the limitations of our approach in several experiments.
Revised: We show the limitations of our approach in several experiments.

Original: Many more validation and verification steps would be need before  real-world deployment to ensure robustness to various unknown situations, corner-cases, adversarial attacks,  biases. 
Revised: Many more validation and verification steps would be crucial before considering its real-world implementation to ensure robustness to various unknown situations, including corner cases, adversarial attacks, and potential biases. A CKNOWLEDGMENTS

Revised: This work was supported by AID Project ACoCaTherm and Hi!Paris.

Revised: This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-AD011011970R1) and (Grant 2022AD011011970R2).

Original: T ABLE OF C ONTENTS - S UPPLEMENTARY M ATERIALS
Revised: T ABLE OF C ONTENTS - S UPPLEMENTARY M ATERIAL A Notations

Original: A Notations

Original: B Implementation details
Revised: B Implementation details

Original: C Discussion on the sparsity
Revised: C Discussion on the sparsity

Original: D Ablation Study
Revised: D Ablation study

Original: E Discussion about OOD criteria
Revised: E Discussion about OOD criteria

Original: F Discussion about the sources of stochasticity
Revised: F Discussion about the sources of stochasticity

Original: G Discussion about the subnetworks
Revised: G Discussion about the subnetworks

Original: H Discussion about the training velocity
Revised: H Discussion about the training velocity

Original: I
Revised: I

Original: Distribution shift
Revised: Distribution shift

Original: J Stabilization of the performance
Revised: J Stabilization of the performance

Original: K On the equivalence between sequential training and Packed-Ensembles
Revised: K On the equivalence between sequential training and Packed-Ensembles

Original: L Using groups is not sufficient to provide equivalent results to Packed-Ensembles
Revised: L Using groups is not sufficient to equal Packed-Ensembles

Original: M Efficiency of the networks trained on ImageNet
Revised: M Efficiency of the networks trained on ImageNet

Original: N Regression
Revised: N Regression

Original: A N OTATIONS
Revised: A N OTATIONS

Original: We summarize the main notations used in the paper in Table
Revised: We summarize the main notations used in the paper in Table

Original: [Table] 
Revised: [Table] [Table]

Revised: The

Original: [Table]
Revised: [Table]

Original: D
Revised: of

Original: [Table]
Revised: [Table]

Original: data samples
Revised: D

Original: [Table]
Revised: [Table]

Original:  the corresponding labels
Revised: data samples and the corresponding labels

Original: [Table]
Revised: [Table]

Original:  index of the current layer, the current subnetwork, and the number of layers
Revised: The index of the current layer, the current subnetwork, and the number of layers

Original: [Table]
Revised: [Table]

Original: activation
Revised: The preactivation feature map [Table] the [Table]

Revised: [Table]

Revised: [Table]

Revised: layer j

Original: [Table]
Revised: [Table]

Original:  constant throughout the network) 
Revised: The activation function (considered constant throughout the network) [Table] The [Table]

Revised: The

Original: [Table]
Revised: [Table]

Original: feature
Revised: channels

Original: [Table]
Revised: [Table]

Original: number of parameters
Revised: the feature maps

Original: [Table]
Revised: [Table]

Original: layer
Revised: layer

Revised: [Table]

Revised: The number of parameters of layer j

Original: [Table]
Revised: [Table]

Original:  batch size of the training procedure
Revised: The batch size of the training procedure

Original: [Table]
Revised: [Table]

Original: The mask corresponding to the layer j of the subnetwork m
Revised: The mask corresponding to the layer j of the subnetwork m

Original: [Table]
Revised: [Table]

Revised: The floor function

Revised: [Table]

Original: The 2D cross-correlation, the convolution and the Hadamard product
Revised: The 2D cross-correlation, the convolution, and the Hadamard product

Original: [Table]
Revised: [Table]

Original: The size of the kernel of the layer j
Revised: The size of the kernel of the layer j

Original: [Table]
Revised: [Table]

Original: The number of subnetworks in an ensemble
Revised: The number of subnetworks in an ensemble

Original: [Table]
Revised: [Table]

Original: The prediction of the subnetwork number m concerning the input x
Revised: The prediction of the subnetwork number m concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The prediction of the ensemble concerning the input x
Revised: The prediction of the ensemble concerning the input x

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: The width-augmentation factor of Packed-Ensembles
Revised: The width-augmentation factor of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The number of subgroups of Packed-Ensembles
Revised: The number of subgroups of Packed-Ensembles

Original: [Table]
Revised: [Table]

Original: The set of weights of the subnetwork m with a width factor α
Revised: The set of weights of the subnetwork m with a width factor α

Original: [Table]
Revised: [Table]

Original: The weights of layer j with γ groups and a width factor α
Revised: The weights of layer j with γ groups and a width factor α

Original: [Table]
Revised: [Table]

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: General Considerations.
Revised: General Considerations.

Original: Our code is implemented in PyTorch (Paszke et al., 2019) using the

Original: PyTorch Lightning framework.

Original: The code will be made publicly available after the anonymity period.

Original: Table 4 summarizes all the hyperparameters used in the paper for CIFAR-10 and CIFAR-100.
Revised: Table 4 summarizes all the hyperparameters used in the paper for CIFAR-10 and CIFAR-100.

Original: In all cases, we use SGD combined with a multistep-learning-rate scheduler multiplying the rate by γ -lr at each milestone.
Revised: In all cases, we use SGD combined with a multistep-learning-rate scheduler multiplying the rate by γ -lr at each milestone.

Original: Note that BatchEnsemble based on ResNet-50 uses a smaller learning rate of 0.08 instead of 0.1 for stability.
Revised: Note that BatchEnsemble based on ResNet50 uses a lower learning rate of 0.08 instead of 0.1 for stability.

Original: The medium data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0 .
Revised: The ”Medium” data augmentation corresponds to a combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0 .

Original: 5 switch probability and using timm’s augmentation classes 3 , with coefficients respectively 0.5 and 0.2.
Revised: 5 switch probability and using timm’s augmentation classes (Wightman, 2019), with coefficients respectively 0.5 and 0.2.

Original: We also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1 in this case.
Revised: In this case, we also use RandAugment (Cubuk et al., 2020) with m = 9 , n = 2 , and mstd = 1 and a label-smoothing (Szegedy et al., 2016) of intensity 0.1.

Original: To ensure that the layers convey sufficient information and are not weakened by groups, we havechosen to set a constant minimum number of channels per group to 64 for all experiments presented in the paper.
Revised: To ensure that the layers convey sufficient information and are not weakened by groups, we have set a constant minimum number of channels per group to 64 for all experiments presented in the paper.

Original: In this case, we choose themaximum number of groups respecting this constraint.
Revised: If the number of channels per group is lower than this threshold, γ is reduced. Moreover, we do not apply subgroups (parameterized by γ ) on the first layer of the network, nor on the first layer of ResNet’s blocks.

Original: Experiments in which this minimum  could play a significant part and lead to confusions are not presented (see for instance PE-( 1 , 4 , 4 ) in Table 5).
Revised: Experiments in which this minimum number of channels could play a significant role and bring confusion are not presented (see, for instance, PE-( 1 , 4 , 4 ) in Table 5).

Original: For ImageNet, we use the A3 procedure from Wightman et al.
Revised: For ImageNet, we use the A3 procedure from Wightman et al.

Original: (2021) for all models.
Revised: (2021) for all models.

Original: Training with the exact A3 procedure was not always possible.
Revised: Training with the exact A3 procedure was not always possible.

Original: Refer to the specific subsection for more details.
Revised: Refer to the specific subsection for more details.

Original: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).
Revised: Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021).

Original: We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).
Revised: We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).

Original: Masksembles.
Revised: Masksembles.

Original: We use the code proposed by (Durasov et al., 2021) 4 .
Revised: We use the code proposed by (Durasov et al., 2021) 3 .

Original: We modified the mask generation function using binary search as proposed by the authors, since it was unable to build masks for ResNet50x4.
Revised: We modified the mask generation function using binary search, as proposed by the authors since it was unable to build masks for ResNet50x4.

Original: We note that the code implies performing batch repeats at the start  the forward passes.
Revised: We note that the code implies performing batch repeats at the start of the forward passes.

Original: All the results regarding this technique are therefore computed with this specification.
Revised: All the results regarding this technique are therefore computed with this specification.

Original: The
Revised: The

Original: ResNet implementations are built using Masksemble2D layers with M = 4 and a scale factor ofafter each convolutions.
Revised: ResNet implementations are built using Masksemble2D layers with M = 4 and a scale factor ofafter each convolution.

Original: BatchEnsemble.
Revised: BatchEnsemble.

Original: For BatchEnsemble, we use two different values for weight decay. Table 4 provides the weight decay corresponding to the shared weights. However, no weight decayis applied to the vectors S and R (which generate the rank-1 matrices).
Revised: For BatchEnsemble, we use two different values for weight decay: table 4 provides the weight decay corresponding to the shared weights but we don’t apply weight decay to the vectors S and R (which generate the rank-1 matrices).

Original: ImageNet.
Revised: ImageNet.

Original: The batch size of Masksembles ResNet-50x4 is reduced to 1120 because of memory constraints.
Revised: The batch size of Masksembles ResNet-50x4 is reduced to 1120 because of memory constraints.

Original: Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence.
Revised: Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence.

Original: C D ISCUSSION ON THE SPARSITY
Revised: C D ISCUSSION ON THE SPARSITY

Original: In this section, we are going to provide an estimation of the expected distance between a dense, fully-connected layer, and a sparse one.
Revised: In this section, we estimate the expected distance between a dense, fully-connected layer and a sparse one.

Original: For simplicity, we are here assuming to operate with a fully-connected layer.
Revised: For simplicity, we are here assuming to operate with a fully-connected layer.

Original: First, let us write our first proposition:
Revised: First, let us write our first proposition:

Original: Proposition C.1.
Revised: Proposition C.1.

Original: Given a fully connected layer j + 1 defined by:
Revised: Given a fully connected layer j + 1 defined by:

Original: [Equation]
Revised: [Equation]

Original: and its approximation defined by:
Revised: and its approximation defined by:

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:
Revised: Under the assumption that the j follows a Gaussian distribution h j ∼ N ( µ j , Σ j ) , where Σ j is the covariance matrix, and µ j the mean vector, the Kullback–Leibler divergence between the layer and its approximation is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜
Revised: where p ∈ [0; 1] is the fraction of the parameters of z j +1 ( c ) included in the approximation ˜

Original: z
Revised: z

Original: j +1 ( c ) .
Revised: j +1 ( c ) .

Original: A plot for (8) is provided in Figure 5.
Revised: A plot for (7) is provided in Figure 5.

Original: Proof.
Revised: Proof.

Original: To prove Prop.
Revised: To prove Prop.

Original: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,
Revised: C.1, we state first that, since h j ( k ) follows a Gaussian distribution, and considering that ω j at inference time is constant and linearly-combined with a gaussian random variable,

Original: z j +1 will be as well gaussian-distributed.
Revised: z j +1 will be as well gaussian-distributed.

Original: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:
Revised: From the property of linearity of expectations, we know that the mean for z j +1 ( c ) is:

Original: [Equation]
Revised: [Equation]

Original: and the variance is:
Revised: and the variance is:

Original: [Equation]
Revised: [Equation]

Original: If we assume Σ( i, k ) = 0 ∀ i ̸
Revised: If we assume Σ( i, k ) = 0 ∀ i ̸

Original: = k , (10) simplifies into:
Revised: = k , (9) simplifies into:

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the case with the mask, similarly as presented in (3):
Revised: Let us now consider the case with the mask, similarly as presented at the end of section 2.1:

Original: [Equation]
Revised: [Equation]

Original: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).
Revised: We assume here that mask j ∼ Ber ( p ) where p is the probability of the Bernoulli (or 1-pruning rate).

Original: In the limit of large C j , we know that ˜
Revised: In the limit of large C j , we know that ˜

Original: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:
Revised: z j +1 ( c ) follows a Gaussian distribution defined by a mean and a variance equal to:

Original: [Equation]
Revised: [Equation]

Original: Hence, we have:
Revised: Hence [Equation] we have:

Original: [Equation]
Revised: [Equation]

Original: In order to assess the dissimilarity between z and ˜ z
Revised: In order to assess the dissimilarity between z and ˜ z

Original: [Equation]
Revised: [Equation]

Original: we can write the Kullback–Leibler divergence:
Revised: we can write the Kullback–Leibler divergence:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Straightforwardly we can write the inequality:
Revised: Straightforwardly we can write the inequality:

Original: [Equation]
Revised: [Equation]

Original: According to (16) we can write:
Revised: According to (15) we can write:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we know that
Revised: Since we know that

Original: [Equation]
Revised: [Equation]

Original: we can also write:
Revised: we can also write:

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] according to: (15)
Revised: Finally, according to: (14)

Original: [Equation]
Revised: [Equation]

Original: finding back (8)
Revised: finding back (7)

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D A BLATION S TUDY
Revised: D A BLATION STUDY

Original: Our algorithm mainly depends on three hyperparameters.
Revised: Our algorithm mainly depends on three hyperparameters.

Original: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.
Revised: M represents the number of subnetworks in the ensemble, α controls the power of representation of the DNN, and γ is an extra parameter that controls the sparsity degree of the DNN.

Original: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.
Revised: To evaluate the sensitivity of Packed-Ensembles to these parameters, we train 5 ResNet-50 on CIFAR-10 similarly to the protocol explained in section 4.1.

Original: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performances, in terms of accuracy and AUPR.
Revised: Figures 6 and 7 show that the more we add subnetworks increasing M , the better the performance, in terms of accuracy and AUPR.

Original: We also note that the results are stable with γ .
Revised: We also note that the results are stable with γ .

Original: Moreover,we notice that the resulting accuracy tends to increase with α until it reaches a plateau.
Revised: Moreover, the resulting accuracy tends to increase with α until it reaches a plateau.

Original: These statement are confirmed by the results in Table 5.
Revised: These statements are confirmed by the results in Table 5.

Original: E D ISCUSSION ABOUT OOD CRITERIA
Revised: E D ISCUSSION ABOUT OOD CRITERIA

Original: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.
Revised: Deep Ensembles (Lakshminarayanan et al., 2017) and Packed-Ensembles are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction.

Original: Similarly to Bayesian Neural Network, one can take the softmax outputs of posterior predictive distribution, which define the MSP = max y i { P ( y i | x , D ) } .
Revised: Similarly to Bayesian Neural Network, one can take the softmax outputs of posterior predictive distribution, which define the MSP = max y i { P ( y i | x , D ) } .

Original: The MSP can also be used for classical DNN, yetin this case, we use the conditional likelihood instead of the posterior distribution.
Revised: The MSP can also be used for classical DNN, yet we use the conditional likelihood instead of the posterior distribution in this case.

Original: One can also use the Maximum Logit (ML)as an uncertainty criterion and the entropy of the posterior predictive distribution asan uncertainty criterion, which is defined by Ent.
Revised: One can also use the Maximum Logit (ML) and the entropy of the posterior predictive distribution as uncertainty criteria, which is defined by Ent.

Original: =
Revised: =

Original: H ( P ( y i | x , D )) with H being the entropy function.
Revised: H ( P ( y i | x , D )) with H being the entropy function.

Original: Another metric is the mutual information between two random variables, which is defined by: MI =H ( P ( y i | x , D ))
Revised: Another metric is the mutual information between two random variables, which is defined by: MI =

Original:  − 1 M M − 1 (cid:80) m 
Revised: H ( P ( y i | x , D )) − 1 M M − 1 (cid:80) m =

Original: =0 H ( P ( y | θ α,m , x )) .
Revised: 0 H ( P ( y | θ α,m , x )) .

Original: It represents a [Figure]
Revised: It represents a measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: measure of the ensemble entropy, which is the entropy of the posterior minus the average entropy over predictions.

Original: The last metric used in active learning is the variation ratio (Beluch et al., 2018) which measures the dispersion of a nominal variable[Table] and is calculated as the proportion of predicted class labels that are not the modal class prediction.
Revised: The last metric, used in active learning, is the variation ratio (Beluch et al., 2018), which measures the dispersion of a nominal variable and is calculated as the proportion of predicted class labels that are not the modal class prediction.

Original: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.
Revised: It is defined by: v = 1 − f i M , where f i is the number of predictions falling into the modal class category.

Original: In Table 6, the results for the different metrics are reported.
Revised: In Table [Table] the results for the different metrics are reported.

Original: We note that ML seems to be the best metric to detect OOD.
Revised: We note that ML seems to be the best metric to detect OOD.

Original: This metric is followed by Ent.
Revised: This metric is followed by Ent.

Original: and then MI .
Revised: and then MI.

Original: Note that v ,which is widely used in active learning, does not seemto be effective in detecting OOD.
Revised: Note that v , widely used in active learning, does not seem effective in detecting OOD samples.

Original: This shows us that it is essential to use a good criterion in addition to good ensembling.
Revised: This shows us that it is essential to use a good criterion in addition to good ensembling.

Original: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY
Revised: F D ISCUSSION ABOUT THE SOURCES OF STOCHASTICITY

Original: As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection.
Revised: As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection.

Original: Three main sources can induce weight diversity and therefore diversity in the function space during the training.
Revised: Three primary sources can induce weight diversity, and therefore diversity in the function space, during the training.

Original: These sources are the initialization of the weights, the composition of the batches and the use of non-deterministic backpropagation algorithms 5 .
Revised: These sources are the initialization of the weights, the composition of the batches, and the use of non-deterministic backpropagation algorithms 4 .

Original: On Table 7, we measure the performance and diversity of Packed-Ensembles trained on CIFAR-100.
Revised: On Table 7, we measure the performance and diversity of PackedEnsembles trained on CIFAR-100.

Original: This diversity is measured by the mutual information and is twofold: we compute the in-distribution mutual information (ID MI ) on the test set of CIFAR-100 and the OOD mutual information (OOD MI ) on SVHN.
Revised: The quantity of diversity is measured by the mutual information and is twofold: we compute the in-distribution mutual information (ID MI ) on the test set of CIFAR-100 and the OOD mutual information (OOD MI ) on SVHN.

Original: of the mean performance of Packed-Ensembles overexperiments in accuracy, NLL, calibration, and OOD detection on CIFAR-100.

Original: Concerning the performance, we compute the accuracy, ECE, and AUPR which are proxies of the quality of this diversity.
Revised: Concerning the performance, we compute the accuracy, ECE, and AUPR, which are proxies of the quality of this diversity.

Original: Results of Table 7 lead to several takeaways.
Revised: Results of Table 7 lead to several takeaways.

Original: First, they hint that there is no clear best set of trivial sources of stochasticity.
Revised: First, they hint that there is no clear best set of trivial sources of stochasticity.

Original: Except for the first (and greyed) line, which corresponds to ensembling completely identical networks (the training being totally deterministic whichis confirmed by the null MI), the results seem equivalent in diversity (viathe mutual information) andin ID/OOD performance.
Revised: Except for the first (and greyed) line, which corresponds to ensembling completely identical networks (the training being totally deterministic, which the null MI confirms), the results seem equivalent in diversity (via mutual information) and ID/OOD performance.

Original: Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity.
Revised: Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity.

Original: It was noted that this effect does not always happen depending on the selected architecture and the precision used ( float16 , or float32 ).
Revised: It was noted that this effect does not always happen depending on the selected architecture and the precision used ( float16 , or float32 ).

Original: Given that there is no emerging best set of stochasticity, wechoose in practice to use the faster non-deterministic backpropagation algorithms as well as different initializations to ensure enough stochasticity and for programming convenience.
Revised: Given that there is no emerging best set of stochasticity, we use the faster non-deterministic backpropagation algorithms and different initializations to ensure enough stochasticity and for programming convenience.

Original: [Table]
Revised: [Table]

Original: G D ISCUSSION ABOUT THE SUBNETWORKS
Revised: G D ISCUSSION ABOUT THE SUBNETWORKS

Original: Thediscussion between the width and depth of deep neural networks is an essential topic of research. Researchers have focused on the correct approaches to increase the depth of DNN [Table] which in turn increases the accuracy.
Revised: The width and depth of deep neural networks are crucial research topics, and researchers strive to determine the best approaches for increasing the depth of DNNs, which can lead to improved accuracy.

Original:  Nguyen et al.
Revised: According to Nguyen et al.

Original: (2020) demonstrate that the width and depth  are connected to the model capacityand enable the DNN to learn block structures which lead to good accuracy.
Revised: (2020), the width and depth of a DNN are connected with its capacity to learn block structures, which can improve accuracy.

Original: Hence, by splitting the width, chances are that we could decrease  the model capacity.
Revised: Therefore, the model’s capacity may decrease if the width is divided.

Original: Deep neural networks are models with huge amounts of parameters, works like the lottery ticket hypothesis (Frankle & Carbin, 2018) suggest that these DNNs are heavily overparameterized.
Revised: Deep neural networks are heavily over-parameterized, as stated by the lottery ticket hypothesis (Frankle & Carbin, 2018).

Original: Thus it is possible to kill up to 80% ofthe neurons  without losing too much performance.
Revised: It suggests that up to 80% of neurons can be removed without significant loss of performance.

Original:  MIMO  builds on this assumption to ensure that each network drives several networks simultaneously.
Revised: The MIMO approach builds on this assumption by allowing multiple networks to be trained simultaneously

Original: However, unlike in our work, the boundaries between the networks are not clearly defined.
Revised: [Table]

Original: Thus, the same neuron can be used for multiple different subnetworks (DNNs) in the same ensemble. In our case, weclearly assign each neuron to a  DNN from the ensemble.
Revised: and neurons may be used by several subnetworks. In our work, however, we assign each neuron to a specific DNN in the ensemble, guaranteeing their independence.

Original: This way, the DNNsdo not get mixed up and can learnan independent representation.
Revised: This way, the DNNs can learn independent representations.

Original: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.
Revised: However, as in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs.

Original: This decomposition may seem crude. However, it allows us to better parallelize Packed-Ensembles for training and inference.
Revised: Although the decomposition may seem crude, it facilitates better parallelization of Packed-Ensembles during training and inference.

Original: To overcome the fact that our networks are not  wide enough if M is too large, wehave added an alpha hyperparameter that can increase the width ofthe subnetworks.
Revised: To address the problem of not having sufficiently wide subnetworks, we added a hyperparameterα - to increase the width of subnetworks.

Original: In Figure 8, we study the influence of the width of the subnetworks.
Revised: In Figure 8, we explore the impact of subnetwork width.

Original: First of all, we notice that the accuracy  increases with the width, while the AUPR looks relatively constant.
Revised: Our observations reveal that the accuracy of the DNN increases with the width while the AUPR remains relatively constant.

Original: This seems to confirm the importance of the alpha parameter inbalancing the width of the DNN.
Revised: This finding suggests that the α is paramount in maintaining a balance in the DNN’s width.

Original: Furthermore, reducing the width  does not appear to reduce the accuracy significantly.
Revised: We also note that reducing the width of the DNN does not significantly impact its accuracy.

Original: Thus, this justifies our choice to split the width of the DNN to produce several subnetworks  since the uncertainty quantification will be constant, and the accuracy will not drop drastically.
Revised: Hence, our decision to split the width of the DNN to create multiple subnetworks is justified since the uncertainty quantification remains unaltered, and the accuracy is not significantly compromised. [Figure] [Table]

Original: Moreover,
Revised: In addition

Original: [Figure]
Revised: [Table]

Revised: α provides an additional degree of freedom to our ensemble

Original: [Table]
Revised: [Table]

Original: the addition of the alpha parameter allows us to add a new degree of freedom to our ensemble and enables it to increase its accuracy
Revised: enabling it to enhance its accuracy. This is a significant advantage

Original: [Table]
Revised: [Table]

Revised: as it allows us to further balance the performance of the ensemble, which can lead to more accurate predictions - and the number of parameters linked to its computational cost.

Original: H D ISCUSSION ABOUT THE TRAINING VELOCITY
Revised: H D ISCUSSION ABOUT THE TRAINING VELOCITY

Original: Our experiments show that grouped convolutions are not as fast as they could theoretically be [Table] and confirm the statements made by many PyTorch and TensorFlow users 6 .
Revised: Our experiments show that grouped convolutions are not as fast as they could theoretically be, and confirm the statements made by many PyTorch and TensorFlow users 5 .

Original: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed
Revised: Following the idea that grouped convolutions are bandwidth-bound, we advise readers to leverage Native Automatic Mixed

Original: Precision (AMP) and cuDNN benchmark flags when training an Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.
Revised: Precision (AMP) and cuDNN benchmark flags when training a Packed-Ensembles to reduce the bandwidth bottleneck compared to the baseline.

Original: AMPwill also divide the VRAM usage by two while yielding equally good results.
Revised: AMP also divides the VRAM usage by two while yielding equally good results.

Original: Future improvements of PyTorch grouped convolutions should help Packed-Ensembles develop its full potential, increasing its current assets.
Revised: Future improvements of PyTorch grouped convolutions should help Packed-Ensembles develop its full potential, increasing its current assets.

Original: We note in Tablethat using float16 , Packed-Ensembles is only 1 .
Revised: We note in Table 8 that using float16 , Packed-Ensembles is only 1 .

Original: 6 × slower than the single model during inference.
Revised: 6 × slower than the single model during inference.

Original: Furthermore, Packed-Ensembles is only 2 .
Revised: Furthermore, Packed-Ensembles is only 2 .

Original: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.
Revised: 3 × slower during training than the single model, making it an efficient model capable of training four models in half the time of a Deep Ensembles.

Original: I D ISTRIBUTION SHIFT
Revised: I D ISTRIBUTION SHIFT

Original: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.
Revised: In this section, we evaluate the robustness of Packed-Ensembles under dataset shift.

Original: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are 5 levels of perturbation called ”severity”, from 1, the weakest to 5, the strongest.
Revised: We use models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks & Dietterich, 2019) to produce CIFAR-100-C. There are five levels of perturbations called ”severity”, from one, the weakest, to five, the strongest.

Original: In real world scenario, distributional shift is crucial as explained by (Ovadia et al., 2019), and it is crucial to study how much a model prediction shifts from the original training data distribution.
Revised: In real-world scenarios, [Figure] distributional shift is crucial, as explained by (Ovadia et al., 2019), and it is critical to study how much a model prediction shifts from the original training data distribution.

Original: Thanks to Figure 9 we notice that[Figure] Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.
Revised: Thanks to Figure 9, we notice that Packed-Ensembles achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.

Original: J S TABILIZATION OF THE PERFORMANCE
Revised: J S TABILIZATION OF THE PERFORMANCE

Original: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute a variance.
Revised: We perform five times each training task on CIFAR-10 and CIFAR-100 to estimate a better value and be able to compute the variance.

Original: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .
Revised: Let us first note that the standard deviation for the single DNN on CIFAR-100 with a ResNet-50 architecture amounts to 0 .

Original: The use of ensemble strategies shrinks the standard variation to 0 .
Revised: Ensemble strategies shrink the standard variation to 0 .

Original: 43% for Deep Ensembles and 0 .
Revised: 43% for Deep Ensembles and 0 .

Original: 19% for Packed-Ensembles.
Revised: 19% for Packed-Ensembles.

Original: Thus it seems that Packed-Ensembles makes DNN predictions more stable in addition to improving accuracy and uncertainty quantification.
Revised: Thus it seems that Packed-Ensembles makes DNN predictions more stable in addition to improving accuracy and uncertainty quantification.

Original: This result is interesting as it seems to contradict Neal et al.
Revised: This result is interesting as it appears to contradict Neal et al.

Original: (2018) who claim that wider DNNs have a smaller variance.
Revised: (2019), who claim that wider DNNs have a smaller variance.

Original: This stability might come from the ensembling.
Revised: This stability might come from the ensembling.

Original: K O N THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES
Revised: K O N THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND P ACKED -E NSEMBLES

Original: The sequential training of Deep Ensembles differs significantly from the training procedure of
Revised: The sequential training of Deep Ensembles differs significantly from the training procedure of

Original: Packed-Ensembles.
Revised: Packed-Ensembles.

Original: The main differences lie in the subnetworks’ batch composition and the best models’ selection.
Revised: The main differences lie in the subnetworks’ batch composition and the best models’ selection.

Original: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.
Revised: Concerning Packed-Ensembles, the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning.

Original: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see
Revised: Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see

Original: Appendix F for more details).
Revised: Appendix F for more details).

Original: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.
Revised: For the selection of models, Packed-Ensembles considers subnetworks as a whole (i.e., maximize the ensemble accuracy on the validation set) and therefore selects the best ensemble at a given epoch.

Original: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.
Revised: On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.

Original: L U SING GROUPS IS NOT SUFFICIENT TO PROVIDE EQUIVALENT RESULTS TO P ACKED -E NSEMBLES
Revised: L U SING GROUPS IS NOT SUFFICIENT TO EQUAL P ACKED -E NSEMBLES

Original: To make sure thatour results cannot be simply explained by the use of groups, we compare PackedEnsembles to a single ResNeXt-50 (32 × 4d) (Xie et al., 2017) in Table 9.
Revised: To make sure that the use of groups cannot simply explain our results, we compare PackedEnsembles to a single ResNeXt-50 (32 × 4d) (Xie et al., 2017) in Table 9.

Original: The ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of eachblock,
Revised: ResNeXt-50 is fairly equivalent to our method but does not propagate groups, only used in the middle layer of each

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  which are therefore not independent.
Revised: block, which are therefore not independent.

Original: The training optimization procedures are the same as well as the data-augmentation strategiesand are all detailed in Appendix B.
Revised: We keep the same training optimization procedures and data-augmentation strategies detailed in Appendix B.

Original: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET
Revised: M E FFICIENCY OF THE NETWORKS TRAINED ON I MAGE N ET

Original: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.
Revised: Table 10 provides the efficiency of the networks trained on ImageNet-1k (see section 4.1.3), in number of parameters and multiply-additions.

Original: PE-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.
Revised: PE-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.

Original: N R EGRESSION
Revised: N R EGRESSION

Original: To generalize our work, we propose to study regression tasks.
Revised: To generalize our work, we propose to study regression tasks.

Original: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.
Revised: We replicate the setting developed by Hern ´ andez-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayanan et al.

Original: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (21) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).
Revised: For the training in the one-dimensional regression setting, we minimize the gaussian NLL (20) using networks with two outputs neurons which estimate the parameters of a heteroscedastic gaussian distribution (Nix & Weigend, 1994; Kendall & Gal, 2017).

Original: One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance.
Revised: One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance.

Original: The ensemble mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ σ θ ( x [Equation] ) 2 ={+
+}
Revised: The ensemble’s mean ¯ µ θ ( x i ) is computed using the empirical mean over the estimators and the variance using the formula of a mixture ¯ σ θ ( x i ) {+
+}

Original: [Equation] ( x [Equation] ) (Lakshminarayanan et al., 2017).
Revised: = M − 1 (cid:80) m (cid:0) σ θ m ( x i ) 2 + µ θ m ( x i ) 2 (cid:1) − ¯ µ θ ( x i ) (Lakshminarayanan et al., 2017).

Original: [Equation]
Revised: [Equation]

Original: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.
Revised: We compare Packed-Ensembles-( 2 , 3 , 1 ) and Deep Ensembles on the UCI datasets in Table 11.

Original: The subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, containing 400 neurons for the bigger Protein dataset and 200 for the others, and a ReLU non-linearity.
Revised: The subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, containing 400 neurons for the more extensive Protein dataset and 200 for the others, and a ReLU non-linearity.

Original: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.
Revised: The results show that Packed-Ensembles and Deep Ensembles provide equivalent results on most datasets.

Original: [Table]
Revised: [Table]