Original: M ORE OR L ESS : W HEN AND H OW TO B UILD  N EURAL N ETWORK E NSEMBLES
Revised: M ORE OR L ESS : W HEN AND H OW TO B UILD C ONVOLUTIONAL N EURAL N ETWORK E NSEMBLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural network models are applied to many tasks with rising complexity and data sizes.
Revised: Convolutional neural network models are applied to many tasks with rising complexity and data sizes.

Original: Researchers and practitioners seek to scale the representational power of these models by adding more parameters.
Revised: Researchers and practitioners seek to scale the representational power of these models by adding more parameters.

Original: Increasing parameters, though, requires additional critical resources such as memory and compute time leading to increased training and inference cost.
Revised: Increasing parameters, though, requires additional critical resources such as memory and compute time leading to increased training and inference cost.

Original: Thus a consistent challenge is to obtain as high as possible accuracy within a parameter budget.
Revised: Thus a consistent challenge is to obtain as high as possible accuracy within a parameter budget.

Original: As neural network designers navigate this complex landscape, they are guided by conventional wisdom.
Revised: As neural network designers navigate this complex landscape, they are guided by conventional wisdom.

Original: This comes from empirical studies of the design space.
Revised: This comes from empirical studies of the design space.

Original: We identify a critical part of this design space that is not well-understood: That is how to decide between the alternatives of expanding a single  network model or increasing the number of networks and using them together in an ensemble.
Revised: We identify a critical part of this design space that is not well-understood: That is how to decide between the alternatives of expanding a single convolutional network model or increasing the number of networks and using them together in an ensemble.

Original: We study this question in detail through extensive experimentation across various network architectures and data sets.
Revised: We study this question in detail through extensive experimentation across various network architectures and data sets.

Original: Crucially, we provide a robust assessment, missing in previous studies, by ﬁxing the number of parameters.
Revised: Crucially, we provide a robust assessment, missing in previous studies, by ﬁxing the number of parameters.

Original: We also consider a holistic set of metrics such as training time, inference time, and memory usage.
Revised: We also consider a holistic set of metrics such as training time, inference time, and memory usage.

Original: Contrary to conventional wisdom, we show that when we perform a holistic and robust assessment, we uncover a wide design space, where ensembles not only provide better accuracy but also train faster, and deploy at a speed comparable to single convolutional networks with the same total number of parameters.
Revised: Contrary to conventional wisdom, we show that when we perform a holistic and robust assessment, we uncover a wide design space, where ensembles not only provide better accuracy but also train faster, and deploy at a speed comparable to single convolutional networks with the same total number of parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scaling capacity of deep learning models.
Revised: Scaling capacity of deep learning models.

Original: Convolutional neural network models are becoming as accurate as humans on perceptual tasks and are now used to discover drugs, compress data, and automate gameplay (Pouyanfar et al., 2018).
Revised: Convolutional neural network models are becoming as accurate as humans on perceptual tasks and are now used to discover drugs, compress data, and automate gameplay (Pouyanfar et al., 2018).

Original: These models have grown rapidly in size, driven by two major trends: First, a rise in data complexity and sizes, which pushes for models with large number of parameters and layers (Szegedy et al., 2015;
Revised: These models have grown rapidly in size, driven by two major trends: First, a rise in data complexity and sizes, which pushes for models with large number of parameters and layers (Szegedy et al., 2015;

Original: He et al., 2016; Huang et al., 2016; Shazeer et al., 2017).
Revised: He et al., 2016; Huang et al., 2016; Shazeer et al., 2017).

Original: Second, the need for better accuracy in critical applications – such as self-driving cars and medical diagnosis – where, to achieve a target accuracy, model sizes are increased routinely (Grzywaczewski, 2017).
Revised: Second, the need for better accuracy in critical applications – such as self-driving cars and medical diagnosis – where, to achieve a target accuracy, model sizes are increased routinely (Grzywaczewski, 2017).

Original: This is especially pronounced in the areas of computer vision and natural language processing:
Revised: This is especially pronounced in the areas of computer vision and natural language processing:

Original: Model sizes are three orders of magnitude larger than what they were just three years ago (Sanh et al., 2019).
Revised: Model sizes are three orders of magnitude larger than what they were just three years ago (Sanh et al., 2019).

Original: When we increase model sizes, we also increase the time, computation, and memory needed to train and deploy such models.
Revised: When we increase model sizes, we also increase the time, computation, and memory needed to train and deploy such models.

Original: It is a consistent challenge to synthesize or design models that maximize accuracy without requiring tons of resources (Lee et al., 2015b; Huang and Liu, 2017).
Revised: It is a consistent challenge to synthesize or design models that maximize accuracy without requiring tons of resources (Lee et al., 2015; Huang and Liu, 2017).

Original: The question we explore in this paper is: Given a set of available resources, which maps to an amount of parameters we can use, how to design a convolutional neural network model?
Revised: The question we explore in this paper is: Given a set of available resources, which maps to an amount of parameters we can use, how to design a convolutional neural network model?

Original: The holistic design space is very complex.
Revised: The holistic design space is very complex.

Original: Designers of convolutional neural network models navigate a complex design landscape in order to address this question: To start off, they decide on a network architecture to use in their model.
Revised: Designers of convolutional neural network models navigate a complex design landscape in order to address this question: To start off, they decide on a network architecture to use in their model.

Original: Then, they have to consider whether to use a single network or build an ensemble model with multiple networks.
Revised: Then, they have to consider whether to use a single network or build an ensemble model with multiple networks.

Original: Additionally, they have to decide how many neural networks to use as well as their individual designs i.e., to ﬁnd a desirable conﬁguration of depth, width, and number of networks in their model.
Revised: Additionally, they have to decide how many neural networks to use as well as their individual designs i.e., to ﬁnd a desirable conﬁguration of depth, width, and number of networks in their model.

Original: Modern applications with diverse requirements further complicate these decisions as what is desirable varies.
Revised: Modern applications with diverse requirements further complicate these decisions as what is desirable varies.

Original: Facebook, for instance, requires convolutional neural network models that strike speciﬁc tradeoffs between accuracy and inference time across 250 different types of smartphones (Wu et al., 2019).
Revised: Facebook, for instance, requires convolutional neural network models that strike speciﬁc tradeoffs between accuracy and inference time across 250 different types of smartphones (Wu et al., 2019).

Original: As a result, not just accuracy but a diversity of metrics – such as inference time and memory usage – inform whether a model gets used (Sze et al., 2017b; Wu et al., 2019).
Revised: As a result, not just accuracy but a diversity of metrics – such as inference time and memory usage – inform whether a model gets used (Sze et al., 2017b; Wu et al., 2019).

Original: Scattered conventional wisdom.
Revised: Scattered conventional wisdom.

Original: There are bits and pieces of scattered conventional wisdom to guide a neural network designer.
Revised: There are bits and pieces of scattered conventional wisdom to guide a neural network designer.

Original: These take the form of various empirical studies that demonstrate how depth and width in a single neural network model relate to accuracy and training time.
Revised: These take the form of various empirical studies that demonstrate how depth and width in a single neural network model relate to accuracy and training time.

Original: First, it is generally known that deeper and wider networks can improve accuracy.
Revised: First, it is generally known that deeper and wider networks can improve accuracy.

Original: In fact, recent convolutional architectures – such as ResNets and DenseNets – are designed precisely to enable this outcome (He et al., 2016; Huang and Liu, 2017; Huang et al., 2017).
Revised: In fact, recent convolutional architectures – such as ResNets and DenseNets – are designed precisely to enable this outcome (He et al., 2016; Huang and Liu, 2017; Huang et al., 2017).

Original: The caveat with beeﬁng up a neural network is that accuracy runs into diminishing returns as we continue to add more layers or widen existing ones (Coates et al., 2011; Dauphin and Bengio, 2013).
Revised: The caveat with beeﬁng up a neural network is that accuracy runs into diminishing returns as we continue to add more layers or widen existing ones (Coates et al., 2011; Dauphin and Bengio, 2013).

Original: On the other hand, increasing the number of networks in the model, i.e., building ensembles, is considered to be a relatively robust but expensive approach to improve accuracy as ensemble models train and deploy k networks instead of one (Russakovsky et al., 2015; Wasay et al., 2020).
Revised: On the other hand, increasing the number of networks in the model, i.e., building ensembles, is considered to be a relatively robust but expensive approach to improve accuracy as ensemble models train and deploy k networks instead of one (Russakovsky et al., 2015; Wasay et al., 2020).

Original: The general consensus is to use ensembles when the goal is to achieve high accuracy without much regards to training cost, inference time, and memory usage e.g., competitions such as COCO and ImageNet (Lee et al., 2015a; Russakovsky et al., 2015; Huang et al., 2017; Ju et al., 2017).
Revised: The general consensus is to use ensembles when the goal is to achieve high accuracy without much regards to training cost, inference time, and memory usage e.g., competitions such as COCO and ImageNet (Lee et al., 2015; Russakovsky et al., 2015; Huang et al., 2017; Ju et al., 2017).

Original: All these studies, however, exist in silos.
Revised: All these studies, however, exist in silos.

Original: Any form of cross-comparison is not possible as they use different data sets, network architectures, and hardware.
Revised: Any form of cross-comparison is not possible as they use different data sets, network architectures, and hardware.

Original: Lack of a robust and holistic assessment.
Revised: Lack of a robust and holistic assessment.

Original: Most empirical studies operate within the conﬁnes of a single convolutional network and do not consider the dimension of ensemble models.
Revised: Most empirical studies operate within the conﬁnes of a single convolutional network and do not consider the dimension of ensemble models.

Original: Those thatdo compare with ensembles,  do so unfairly comparing ensembles with k networks against a single model that contains one such network.
Revised: Those that compare with ensembles, mostly do so unfairly comparing ensembles with k networks against a single model that contains one such network (Lee et al., 2015; Russakovsky et al., 2015; Huang et al., 2017; Ju et al., 2017).

Original: The problem with this  is that the amount of parameters is not comparable, typically being k times more for ensembles, where k is the number of networks in the ensemble.
Revised: To rectify this, there are recent studies that make this comparison under a parameter budget. The major conclusion is that beyond a certain budget, ensembles can provide better accuracy than single networks (Chirkova et al., 2020; Kondratyuk et al., 2020).

Original: In this way, such an analysis is an apples-to-oranges comparison and, as a result, it is not helpful in isolating the fundamental properties of ensembles versus single models. In addition, past studies consider only accuracy and training cost.
Revised: This work, however, considers only the metric of generalization accuracy and explores a very limited design space – two different classes of convolutional architectures with a single depth.

Original: However, a holistic analysis needs to include also inference cost and memory usage, both of which are metricsthat in practical settings are critical for  applications (Sze et al., 2017a; Wu et al., 2019).
Revised: However, a holistic analysis needs to include resource-related metrics such as training time, inference cost, and memory usage. All these metrics are critical for practical applications (Sze et al., 2017a; Wu et al., 2019).

Original: The primary reason these metrics are excluded from the analysis is that the expectation is for an ensemble model to be k times slower for inference and k times more memory hungry compared to a single model.
Revised: Furthermore, to provide reliable guidance to a model designer, a robust comparison needs to consider a range of architectures and model sizes with various conﬁgurations of depth and width. This is critical especially because varying just the width of convolutional networks in isolation, as done by recent studies (Chirkova et al., 2020; Kondratyuk et al., 2020), is known to be far less effective to improve accuracy (Eigen et al., 2013; Ba and Caruana, 2014).

Original: Ensembles vs. single networks.
Revised: Ensembles vs. single networks.

Original: In this paper, we bridge this gap in our understanding of the design space between single network models and ensemble models in a fair and holistic way.
Revised: In this paper, we bridge this gap in our understanding of the design space between single network models and ensemble models in a fair and holistic way.

Original: Given modern applications with various requirements and constraints, should we train and deploy a convolutional model with a single network or one that contains an ensemble of networks?
Revised: Given modern applications with various requirements and constraints, should we train and deploy a convolutional model with a single network or one that contains an ensemble of networks?

Original: How should one approach the design of networks within an ensemble?
Revised: How should one approach the design of networks within an ensemble?

Original: As these constraints and requirements evolve, should we switch between these alternatives, why, and when?
Revised: As these constraints and requirements evolve, should we switch between these alternatives, why, and when?

Original: Method.
Revised: Method.

Original: We introduce the following methodology to accurately map the design space.
Revised: We introduce the following methodology to accurately map the design space.

Original: Since there is no robust theoretical framework to consistently analyze the design space and the complex interactions among its many parameters and metrics, we develop a detailed and extensive experimental framework to isolate the impact of the critical design knobs: (i) depth, (ii) width, and (iii) number of networks, on all relevant metrics: (i) accuracy, (ii) training time, (iii) inference time, and (iv) memory usage.
Revised: Since there is no robust theoretical framework to consistently analyze the design space and the complex interactions among its many parameters and metrics, we develop a detailed and extensive experimental framework to isolate the impact of the critical design knobs: (i) depth, (ii) width, and (iii) number of networks, on all relevant metrics: (i) accuracy, (ii) training time, (iii) inference time, and (iv) memory usage.

Original: Crucially the number of parameters is a control knob in our framework and remains ﬁxed across results that can be studied together.
Revised: Crucially the number of parameters is a control knob in our framework and remains ﬁxed across results that can be studied together.

Original: To establish the robustness of our ﬁndings, we experiment across various architectures, data complexities, and classiﬁcation tasks.
Revised: To establish the robustness of our ﬁndings, we experiment across various architectures, data complexities, and classiﬁcation tasks.

Original: We present and analyze data amounting to over one year of GPU runtime.
Revised: We present and analyze data amounting to over one year of GPU runtime.

Original: We also explain trends breaking down metrics into their constituents when necessary.
Revised: We also explain trends breaking down metrics into their constituents when necessary.

Original: Results: The Ensemble Switchover Threshold (EST).
Revised: Results: The Ensemble Switchover Threshold (EST).

Original: (i)
Revised: (i)

Original: Contrary to conventional wisdom, we show that when we make a holistic and robust comparison between single convolutional networks and ensembles of networks, we discover a vast design space where ensembles provide not just better overall accuracy but also train faster compared to a single network.
Revised: Contrary to conventional wisdom, we show that when we make a holistic and robust comparison between single convolutional networks and ensembles of networks, we discover a vast design space where ensembles provide not just better overall accuracy but also train faster compared to a single network.

Original: (ii) Speciﬁcally, we uncover the Ensemble Switchover Threshold (EST).
Revised: (ii) Speciﬁcally, we uncover the Ensemble Switchover Threshold (EST).

Original: This is the amount of resources (measured in terms of number of parameters and training epochs) beyond which ensembles provide superior generalization accuracy to a single model
Revised: This is the amount of resources (measured in terms of number of parameters and training epochs) beyond which ensembles provide superior generalization accuracy to a single model.

Original: (iii
Revised: (iii)

Original: ) We show that EST occurs consistently across variousdata sets and architectures.
Revised: We show that EST occurs consistently across various

Original: iv
Revised: E E

Original: ) We demonstrate that the number of networks in an ensemble and their individual designs determine the
Revised: [Figure]

Original: EST v
Revised: E E

Original: ) Ensembles can also provide comparable inference times for a considerable part of the design space. (vi) We also show that for the same number of
Revised: E (c) Width

Original: [Figure]
Revised: [Figure]

Original: E
Revised: equivalent ensemble (

Original: [Figure]
Revised: [Figure]

Original: Param. : |S| | Depth
Revised: Depth

Original: [Figure]
Revised: [Figure]

Original: | Width
Revised: d t

Original: [Figure]
Revised: [Figure]

Original: Param.
Revised: Param.

Original: : |S| | Depth
Revised: : |S| | Depth: d | Width: w

Original: [Figure]
Revised: Param.

Original:  d | Width [Figure] w’
Revised: : |S| | Depth: d | Width: w’

Original: Param.
Revised: Param.

Original: : |S| | Depth: d’ | Width: w  parameters, ensembles require signiﬁcantly less memory to train and they are cheaper to train using cloud computing resources
Revised: : |S| | Depth: d’ | Width: w data sets and architectures. (iv) We demonstrate that the number of networks in an ensemble and their individual designs determine the EST. (v) Ensembles can also provide comparable inference times for a considerable part of the design space. (vi) We also show that for the same number of parameters, ensembles require signiﬁcantly less memory to train and they are cheaper to train using cloud computing resources.

Original:  Finally, we make available a web-based demo to enable visual exploration of our experimental results: https://evd-demo.github.io .
Revised: (vii) Finally, we make available a web-based demo to enable visual exploration of our experimental results ( https://evd-demo.github.io ) and release the code ( https://github.com/evd-demo/more-or-less ).

Original:  D EFINING THE D ESIGN S PACE
Revised:  D EFINING THE D ESIGN S PACE

Original: The design space we explore consists of single convolutional neural network models and two classes of architecturally-homogenous ensembles.
Revised: The design space we explore consists of single convolutional neural network models and two classes of architecturally-homogenous ensembles.

Original: We ﬁrst describe how we ensure a fair ranking within this design space and then explain the degrees of freedom we explore.
Revised: We ﬁrst describe how we ensure a fair ranking within this design space and then explain the degrees of freedom we explore.

Original: Establishing grounds for fair ranking.
Revised: Establishing grounds for fair ranking.

Original: We rank various single network and ensemble model designs under equivalent resources.
Revised: We rank various single network and ensemble model designs under equivalent resources.

Original: We do this by comparing designs having the same number of parameters.
Revised: We do this by comparing designs having the same number of parameters.

Original: This allows us to separate the quality of the design itself from the amount of resources given to it.
Revised: This allows us to separate the quality of the design itself from the amount of resources given to it.

Original: Crucially, a model designer with a certain amount of resources can use this resourceequivalent ranking to bootstrap their design process.
Revised: Crucially, a model designer with a certain amount of resources can use this resourceequivalent ranking to bootstrap their design process.

Original: We ﬁx the number of parameters because of its two distinctive properties over other resource metrics (that could possibly be ﬁxed) such as training time, inference time, or memory usage: First, the number of parameters of a network is directly proportional to all other resource-related metrics.
Revised: We ﬁx the number of parameters because of its two distinctive properties over other resource metrics (that could possibly be ﬁxed) such as training time, inference time, or memory usage: First, the number of parameters of a network is directly proportional to all other resource-related metrics (Jain et al., 2020; Wasay et al., 2020).

Original: Second, the number of parameters is independent of the hardware or the software platform used and can be computed exactly from a network speciﬁcation.
Revised: Second, the number of parameters is independent of the hardware or the software platform used and can be computed exactly from a network speciﬁcation.

Original: The single network versus ensemble design space.
Revised: The single network versus ensemble design space.

Original: We start with a single convolutional neural network architecture S ( w,d ) from a class of neural network architectures C having width factor w 1 , depth d , and with | S | number of parameters.
Revised: We start with a single convolutional neural network architecture S ( w,d ) from a class of neural network architectures C having width factor w 1 , depth d , and with | S | number of parameters.

Original: We construct an ensemble E =
Revised: We construct an ensemble E =

Original: { E 1 . . .
Revised: { E 1 . . .

Original: E k } such that the total number of parameters it contains and the architecture class of its networks are the same as those of S i.e., E 1 . . .
Revised: E k } such that the total number of parameters it contains and the architecture class of its networks are the same as those of S i.e., E 1 . . .

Original: E k ∈ C and | E 1 |
Revised: E k ∈ C and | E 1 |

Original: + . . .
Revised: + . . .

Original: + | E k | = | S | .
Revised: + | E k | = | S | .

Original: This ensemble is architecturallyhomogenous i.e., all networks have the same architecture E 1 = . . .
Revised: This ensemble is architecturallyhomogenous i.e., all networks have the same architecture E 1 = . . .

Original: =
Revised: =

Original: E k and each ensemble network has | S | /k parameters.
Revised: E k and each ensemble network has | S | /k parameters.

Original: We construct ensembles in this way to reduce the intractably large space 2 of all possible ensembles given a single network to a size that we can feasibly and thoroughly experiment with and reason about.
Revised: We construct ensembles in this way to reduce the intractably large space 2 of all possible ensembles given a single network to a size that we can feasibly and thoroughly experiment with and reason about.

Original: Furthermore, a considerable amount of neural network ensembles introduced in research as well as used in practice are similarly homogenous, for instance, SnapShot Ensembles and Fast
Revised: Furthermore, a considerable amount of neural network ensembles introduced in research as well as used in practice are similarly homogenous, for instance, SnapShot Ensembles and Fast

Original: Geometric Ensembles (Huang et al., 2017; Garipov et al., 2018).
Revised: Geometric Ensembles (Huang et al., 2017; Garipov et al., 2018).

Original: Additionally, our method provides a deterministic procedure of going between single network models and ensembles given a certain amount of parameters.
Revised: Additionally, our method provides a deterministic procedure of going between single network models and ensembles given a certain amount of parameters.

Original: Major sources of diversity in neural network ensembles are random weight initialization and stochastic training, both of which we incorporate in our experiments.
Revised: Major sources of diversity in neural network ensembles are random weight initialization and stochastic training, both of which we incorporate in our experiments.

Original: Depth-equivalent and width-equivalent ensembles.
Revised: Depth-equivalent and width-equivalent ensembles.

Original: Convolutional neural network architectures are determined by two design knobs – the depth and the width factor.
Revised: Convolutional neural network architectures are determined by two design knobs – the depth and the width factor.

Original: Corresponding to these two design knobs, we create two classes of ensembles: depth-equivalent ensembles and width-equivalent [Table] [Table] [Table]
Revised: Corresponding to these two design knobs, we create two classes of ensembles: depth-equivalent ensembles and width-equivalent ensembles.

Original: ensembles.

Original: These are depicted in Figure 1:
Revised: These are depicted in Figure 1:

Original: In depth-equivalent ensembles the depth of the individual ensemble networks is the same as S (i.e., d ) and the width factor is set to the highest possible value (i.e., w (cid:48) ) without exceeding the parameter budget of | S | .
Revised: In depth-equivalent ensembles the depth of the individual ensemble networks is the same as S (i.e., d ) and the width factor is set to the highest possible value (i.e., w (cid:48) ) without exceeding the parameter budget of | S | .

Original: In width-equivalent ensembles, on the other hand, the width factor is conserved across all ensemble networks (i.e., w ) and the depth is modulated to the highest possible value (i.e., d (cid:48) ) without exceeding | S | :w (cid:48) [Equation] k [Equation]
Revised: In width-equivalent ensembles, on the other hand, the width factor is conserved across all ensemble networks (i.e., w ) and the depth is modulated to the highest possible value (i.e., d (cid:48) ) without exceeding | S | :

Original: E

Original: [Equation]

Original: w (cid:48)

Original: [Equation]

Original: d

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: It follows from the above deﬁnition that neural networks in depth-equivalent ensembles have higher depth than those in width-equivalent ensembles.
Revised: It follows from the above deﬁnition that neural networks in depth-equivalent ensembles have higher depth than those in width-equivalent ensembles.

Original: Width-equivalent ensembles contain wider neural networks than their depth-equivalent counterparts.
Revised: Width-equivalent ensembles contain wider neural networks than their depth-equivalent counterparts.

Original: In this way, we isolate and study the effect of depth and width on ensemble accuracy and resource requirement.
Revised: In this way, we isolate and study the effect of depth and width on ensemble accuracy and resource requirement.

Original: Overall, our design space spans across three classes of convolutional neural network designs: (i) single network models, (ii) width-equivalent ensembles, and (iii) depth-equivalent ensembles.
Revised: Overall, our design space spans across three classes of convolutional neural network designs: (i) single network models, (ii) width-equivalent ensembles, and (iii) depth-equivalent ensembles.

Original: Every class contains several model designs that are instantiated by the four-tuple { w, d, | S | , C } .
Revised: Every class contains several model designs that are instantiated by the four-tuple { w, d, | S | , C } .

Original: We design and conduct experiments across various conﬁgurations of these four-tuples.
Revised: We design and conduct experiments across various conﬁgurations of these four-tuples.

Original:  E XPERIMENTATION
Revised:  E XPERIMENTATION

Original: Datasets and architectures.
Revised: Datasets and architectures.

Original: We include widely-used and state-of-the-art convolutional neural network architectures in our study: VGGNets, ResNets, DenseNets, and Wide ResNets
Revised: We include widely-used and state-of-the-art convolutional neural network architectures in our study: VGGNets, ResNets, DenseNets, and Wide ResNets

Original: (He at al., 2015; Zagoruyko and Komodakis, 2016; Huang and Liu, 2017).
Revised: (He at al., 2015; Zagoruyko and Komodakis, 2016; Huang and Liu, 2017).

Original: We also experiment on data sets with varying complexities and sizes: SVHN, CIFAR-10, CIFAR-100,and Tiny ImageNet (Krizhevsky, 2009; Netzer et al., 2011; Russakovsky et al., 2015).
Revised: We also experiment on data sets with varying complexities and sizes: SVHN, CIFAR-10, CIFAR-100, Tiny ImageNet, and downsampled ImageNet-1k (Krizhevsky, 2009; Netzer et al., 2011; Russakovsky et al., 2015; Cubuk et al., 2020).

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We evaluate single network and ensemble designs across ﬁve metrics: (i) generalization accuracy, (ii) training time per epoch, (iii) time to accuracy, (iv) inference time, and (v) memory usage.
Revised: We evaluate single network and ensemble designs across ﬁve metrics: (i) generalization accuracy, (ii) training time per epoch, (iii) time to accuracy, (iv) inference time, and (v) memory usage.

Original: These metrics when considered together provide a holistic picture of the quality as well as the practicality of various designs.
Revised: These metrics when considered together provide a holistic picture of the quality as well as the practicality of various designs.

Revised: The time to accuracy metric compares in a black box way the time taken by any two models to arrive at a speciﬁed accuracy and has been extensively used to compare the end-to-end performance of deep learning models (Coleman et al., 2019).

Original: Experimental setup.
Revised: Experimental setup.

Original: For all architectures, we adopt hyperparameters (including number of training epochs) from their respective papers.
Revised: For all architectures, we adopt hyperparameters (including number of training epochs) from their respective papers.

Original: We experiment with various model sizes spanning the complete range of model sizes (in terms of number of parameters) reported in literature.
Revised: We experiment with various model sizes spanning the complete range of model sizes (in terms of number of parameters) reported in literature.

Original: Table A in the appendix summarizes these training details.
Revised: Table A in the appendix summarizes these training details.

Original: We also ensure that we reach accuracies reported in literature for all architectures and sizes in our experiments.
Revised: We also ensure that we reach accuracies reported in literature for all architectures and sizes in our experiments.

Original: We implement our experimental framework in PyTorch.
Revised: We implement our experimental framework in PyTorch.

Original: We use an Nvidia V100 GPU to run all our experiments.
Revised: We use an Nvidia V100 GPU to run all our experiments.

Original:  E NSEMBLES O UTPERFORM S INGLE N ETWORK M ODELS A FTER A L OW TO M ODERATE P ARAMETER T HRESHOLD
Revised:  E NSEMBLES O UTPERFORM S INGLE N ETWORK M ODELS A FTER A L OW TO M ODERATE P ARAMETER T HRESHOLD

Original: We observe that after a certain resource threshold, both classes of ensembles – depth- and widthequivalent – outperform single network models.
Revised: We observe that after a certain resource threshold, both classes of ensembles – depth- and widthequivalent – outperform single network models.

Original: We call this threshold the Ensemble Switchover
Revised: We call this threshold the Ensemble Switchover

Original: Threshold (EST).
Revised: Threshold (EST).

Original: Beyond the EST, ensemble models achieve 1 to 3 percent lower absolute test error rates (across various architectures and data sets) when compared with single network models having the same number of parameters.
Revised: Beyond the EST, ensemble models achieve 1 to 3 percent lower absolute test error rates (across various architectures and data sets) when compared with single network models having the same number of parameters.

Original: To the best of our knowledge, this is the ﬁrst observation of this phenomenon.
Revised: To the best of our knowledge, this is the ﬁrst observation of this phenomenon.

Original: The EST appears consistently across a wide range of data sets and architectures (Figure 2(a) through Figure 2(f)) as well as ensemble sizes (Figure 3(a) through Figure 3(c) and appendix Figure B(a) through Figure
Revised: The EST appears consistently across a wide range of data sets and architectures (Figure 2(a) through Figure 2(f)) as well as ensemble sizes (Figure 4(a) through Figure 4(c) and appendix Figure B(a) through Figure

Original: B(c)).
Revised: B(c)).

Original: In these ﬁgures, we use discrete heat maps to visualize which of the three design classes – single network models (single), depth-equivalent ensembles (deq), and width-equivalent ensembles (weq) – dominates in terms of generalization accuracy for a given resource budget.
Revised: In these ﬁgures, we use discrete heat maps to visualize which of the three design classes – [Table] [Figure] [Table]

Original: This resource budget takes the concrete form of number of parameters (on the x-axis) and number of epochs (on the y-axis).
Revised: [Figure] [Table] [Figure] [Table]

Original: We also mark areas, where both classes of ensembles outperform single network models.
Revised: [Figure]

Original: For all experiments, Figure A in the appendix shows the test error rates achieved by various models.
Revised: [Table] [Figure] [Figure]

Original: The occurrence of EST both expands and questions the general consensus on the relative effectiveness of ensemble versus single network models. First, even when allocated the same amount
Revised: single network models (single), depth-equivalent ensembles (deq), and width-equivalent ensembles (weq) – dominates in terms of generalization accuracy for a given resource budget.

Original: [Table] [Figure] [Table] [Figure]
Revised: This resource budget takes the concrete form of number of parameters (on the x-axis) and number of epochs (on the y-axis).

Original: [Table] [Figure]
Revised: We also mark areas, where both classes of ensembles outperform single network models.

Original: [Table] [Figure] [Table] [Figure]
Revised: Figure 3 shows the test error rates achieved on various data sets for DenseNet models.

Original: [Table] [Figure] [Table] [Figure]
Revised: We present this metric for the rest of the architectures in Figure A in the appendix.

Original: [Table] [Figure] [Table] [Figure]
Revised: The occurrence of EST both expands and questions the general consensus on the relative effectiveness of ensemble versus single network models.

Original: [Table] [Figure] of resources , ensemble models still outperform single network models.
Revised: First, even when allocated the same amount of resources, ensemble models still outperform single network models.

Original: This expands upon past empirical studies that only show how a k -network ensemble is more accurate than any of the single network models that it contains (Lee et al., 2015a; Huang et al., 2017).
Revised: This expands upon past empirical studies that only show how a k -network ensemble is more accurate than any of the single network models that it contains (Lee et al., 2015; Huang et al., 2017).

Original: Second, the EST occurs in low- to moderate-resource settings.
Revised: Second, the EST occurs in low- to moderate-resource settings.

Original: For instance, in all of our experiments, we observe the EST at the 1M to 1.5M parameter range 3 and after no later than half of the training epochs.
Revised: For instance, in all of our experiments, we observe the EST at the 1M to 1.5M parameter range 3 and after no later than half of the training epochs.

Original: This challenges the widespread notion that ensembles of neural networks are useful only when we have tons of resources at our disposal (Lee et al., 2015a; Ju et al., 2017).
Revised: This challenges the widespread notion that ensembles of neural networks are useful only when we have tons of resources at our disposal (Lee et al., 2015; Ju et al., 2017).

Original: Overall, our results indicate that ensembles of convolutional models are preferable to single network models for a much wider range of use cases than previously understood. 
Revised: Overall, our results indicate that ensembles of convolutional models are preferable to single network models for a much wider range of use cases than previously understood. On superior generalization of ensembles under a parameter budget.

Original: We parse out how the data complexity and the composition of the ensemble networks affect the EST and in turn the ranking of the three design classes:
Revised: Our interpretation for why ensembles outperform single networks on a parameter budget is that as we keep on increasing the size of a single convolutional neural network we run into diminishing returns and the generalization capability of a single network hits a plateau. In past analysis of single networks, this effect has been independently investigated by Eigen et al. and Dauphin and Bengio (2013). With ensembles, when the generalization accuracy of the single network starts showing diminishing returns, the corresponding width-equivalent and depth-equivalent ensembles have smaller networks with 1 /k as many parameters (assuming the parameters are spread equally along k ensemble networks).

Revised: These individual networks in the ensemble are affected less by the plateau at this point because they have 1 /k the parameters.

Revised: Thus, utilizing these networks in an ensemble leads to better generalization

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: deq only weq only both ens

Revised: [Table]

Revised: deq only weq only both ens.

Revised: single deq only weq only both ens.

Revised: [Table]

Revised: [Figure]

Revised: e r

Revised: o f

Revised: e po

Revised: c h s

Revised: [Figure]

Revised: accuracy overall because they do not hit the diminishing returns threshold while still being able to utilize the known properties that ensembles provide: (i)

Revised: They enrich the space of hypotheses that are considered by the base model class (ii) By averaging over various models, ensembles reduce the variance of base models, smoothing out variations due to initialization and the learning process (Lee et al., 2015).

Revised: Now, we parse out how the data complexity and the composition of the ensemble networks affect the

Revised: EST and in turn the ranking of the three design classes:

Original: Ensembles are even more effective for more complex data sets.
Revised: Ensembles are even more effective for more complex data sets.

Original: We observe that the EST shifts closer to the origin as the complexity of the training data set increases.
Revised: We observe that the EST shifts closer to the origin as the complexity of the training data set increases.

Original: This can be seen in Figure 2(a) through 2(c) where we train DenseNets on progressively more complex data sets (CIFAR-10,
Revised: This can be seen in Figure 2(a) through 2(c) where we train DenseNets on progressively more complex data sets (CIFAR-10,

Original: CIFAR-100, and Tiny ImageNet).
Revised: CIFAR-100, and Tiny ImageNet).

Original: This indicates that ensemble models are preferable to single models, when training on more complex data sets for an even wider range of available resources.
Revised: This indicates that ensemble models are preferable to single models, when training on more complex data sets for an even wider range of available resources.

Original: This observation again expands the utility of ensembles.
Revised: This observation again expands the utility of ensembles.

Original: There is theoretical as well as empirical work establishing that ensembles do better for complex data (Bonab and Can, 2017; Huang et al., 2017).
Revised: There is theoretical as well as empirical work establishing that ensembles do better for complex data (Bonab and Can, 2017; Huang et al., 2017).

Original: We, however, establish this phenomenon in the resource-equivalent setting as opposed to past studies that do so for ensembles and single networks with drastically different number of parameters.
Revised: We, however, establish this phenomenon in the resource-equivalent setting as opposed to past studies that do so for ensembles and single networks with drastically different number of parameters.

Original: Large ensembles are effective under a large parameter budget.
Revised: Large ensembles are effective under a large parameter budget.

Original: As we increase the number of networks k within an ensemble without increasing the parameter budget, the overall accuracy of ensemble designs diminishes pushing the EST to a higher resource limit.
Revised: As we increase the number of networks k within an ensemble without increasing the parameter budget, the overall accuracy of ensemble designs diminishes pushing the EST to a higher resource limit.

Original: Figure 3 demonstrates this phenomenon for DenseNets and Figure B in the appendix shows it for ResNets.
Revised: Figure 4 demonstrates this phenomenon for DenseNets and Figure B in the appendix shows it for ResNets.

Original: For instance, for DenseNet models, the EST moves from the 1 .
Revised: For instance, for DenseNet models, the EST moves from the 1 .

Original: M range for k = 4 to the 3 M range for k= [Figure] [Table] [Figure]
Revised: M range for k = 4 to the 3 M range for k

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: =

Original: [Figure]

Original: [Table] [Figure] [Figure] and, then, to the 5 M range for k = 8 .
Revised: and, then, to the 5 M range for k = 8 .

Original: This can be explained by looking at individual accuracies of ensemble networks.
Revised: This can be explained by looking at individual accuracies of ensemble networks.

Original: Figure C in the appendix shows test error rates of the ensemble as a whole as well as the average test error rates of individual ensemble networks.
Revised: Figure C in the appendix shows test error rates of the ensemble as a whole as well as the average test error rates of individual ensemble networks.

Original: We observe that as we increase the number of networks k , their individual test error rates (shown as dotted lines) increase along with the ensemble accuracy.
Revised: We observe that as we increase the number of networks k , their individual test error rates (shown as dotted lines) increase along with the ensemble accuracy.

Original: This happens because the size of individual networks goes down (as we keep a ﬁxed parameter budget). 
Revised: This happens because the size of individual networks goes down (as we keep a ﬁxed parameter budget). This implies that ensembles of larger size are desirable over those [Figure]

Original: This implies that ensembles of larger size are desirable over those of smaller sizes only when we have a sufﬁcient parameter budget to assign to every single network in the model.
Revised: [Figure] [Figure] of smaller sizes only when we have a sufﬁcient parameter budget to assign to every single network in the model.

Original: As opposed to previous work, our experiments decouple the parameter budget from the number of networks in the model.
Revised: As opposed to previous work, our experiments decouple the parameter budget from the number of networks in the model.

Original: We discover that just increasing the ensemble size without increasing the total number of parameters does not result in improved accuracy.
Revised: We discover that just increasing the ensemble size without increasing the total number of parameters does not result in improved accuracy.

Original: Depth-equivalent ensembles outperform width-equivalent ensembles.
Revised: Depth-equivalent ensembles outperform width-equivalent ensembles.

Original: We observe that depthequivalent ensembles are overall more accurate than width-equivalent ensembles (as shown in Figure A in the appendix).
Revised: We observe that depthequivalent ensembles are overall more accurate than width-equivalent ensembles (as shown in Figure 3 and appendix Figure A).

Original: They also consistently demonstrate EST at a lower resource range.
Revised: They also consistently demonstrate EST at a lower resource range.

Original: This can be explained by the fact that modern convolutional neural network architectures are designed to provide better accuracy with increasing depth.
Revised: This can be explained by the fact that modern convolutional neural network architectures are designed to provide better accuracy with increasing depth.

Original: Depth-equivalent ensembles have deeper ensemble networks with better individual accuracy.
Revised: Depth-equivalent ensembles have deeper ensemble networks with better individual accuracy.

Original: When used together in an ensemble they also provide better ensemble accuracy.
Revised: When used together in an ensemble they also provide better ensemble accuracy.

Original: When designing ensemble models for high accuracy, deeper networks are preferable to wider networks.
Revised: When designing ensemble models for high accuracy, deeper networks are preferable to wider networks.

Original: Our observation also corroborates past empirical studies which show that deeper networks provide better accuracy keeping all other factors constant (Eigen et al., 2013; Urban et al., 2016).
Revised: Our observation also corroborates past empirical studies which show that deeper networks provide better accuracy keeping all other factors constant (Eigen et al., 2013; Urban et al., 2016).

Original: This fact holds also for ensembles.
Revised: This fact holds also for ensembles.

Revised: EST vs. Memory Split Advantage.

Revised: For a limited part of the design space, recent work has observed the existence of a parameter limit beyond which depth-equivalent ensembles outperform single networks.

Revised: This is termed as the Memory Split Advantage or the MSA (Kondratyuk et al., 2020).

Revised: The EST, however, is not deﬁned just with respect to number of parameters but also the number of training epochs, inference cost, and memory usage as by only looking at these metrics in conjunction, we can get a complete picture of the relative effectiveness of ensembles vs. single networks.

Revised: In this way, the EST subsumes the MSA and we also verify the MSA across a much larger design space than has been done before.

Original:  E NSEMBLES T RAIN F ASTER AND P ROVIDE C OMPARABLE I NFERENCE T IME
Revised:  E NSEMBLES T RAIN F ASTER AND P ROVIDE C OMPARABLE I NFERENCE T IME

Original: First, we analyze the training time.
Revised: First, we analyze the training time.

Original: Despite taking longer per epoch, both ensemble classes achieve the accuracy of single network models signiﬁcantly faster for a considerable part of the design space (e.g., 1 . 2 × to 5 × faster across our experiments).
Revised: Despite taking longer per epoch, both ensemble classes achieve the accuracy of single network models signiﬁcantly faster for a considerable part of the design space (e.g., 1 . 2 × to 5 × faster across our experiments).

Original: This happens after the EST has been reached i.e., when ensemble designs can provide better accuracy, they can also do so faster than single network models.
Revised: This happens after the EST has been reached i.e., when ensemble designs can provide better accuracy, they can also do so faster than single network models.

Original: This can be seen in Figure 4.
Revised: This can be seen in Figure 5.

Original: Here, we plot the total training time needed for any of the three 
Revised: Here, we plot the total training time needed for any of the three [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: i n f e r e n ce ti m e p e r
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: i m a g e ( m s )
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: design classes to achieve the maximum accuracy provided by single network models under the same parameter budget.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Original: Figure 5 shows the corresponding training time per epoch.
Revised: m o r y u s a g e ( G B ) [Figure] | 52 . 59 8 | 58 | 54 .

Revised: 38 12 | 52 | 108 .

Revised: 12 | 58 | 122 .

Revised:     single weq deq width factor | depth | number of parameters (M) m a x . m

Revised: e m o r y u s a g e ( G B ) (c) Wide ResNets CIFAR-10 (k=4) design classes to achieve the maximum accuracy provided by single network models under the same parameter budget.

Revised: Figure 6 shows the corresponding training time per epoch.

Original: The combined depth determines per epoch training time.
Revised: The combined depth determines per epoch training time.

Original: We observe that both classes of ensembles, on average, take longer to train per epoch as compared to single network models as they train k networks instead of one.
Revised: We observe that both classes of ensembles, on average, take longer to train per epoch as compared to single network models as they train k networks instead of one.

Original: How much more time ensembles take per epoch depends heavily on the design of the ensemble networks: This additional time is negligible for width-equivalent ensembles whereas, for depth-equivalent ensembles, it results in 2 × more expensive per epoch training.
Revised: How much more time ensembles take per epoch depends heavily on the design of the ensemble networks: This additional time is negligible for width-equivalent ensembles whereas, for depth-equivalent ensembles, it results in 2 × more expensive per epoch training.

Original: This trend can be explained by how the training time per epoch scales with respect to the width and depth of convolutional neural network models.
Revised: This trend can be explained by how the training time per epoch scales with respect to the width and depth of convolutional neural network models.

Original: This ultimately connects to GPU capabilities.
Revised: This ultimately connects to GPU capabilities.

Original: We break down the training time per epoch of all designs into two constituents: time spent per layer and number of layers.
Revised: We break down the training time per epoch of all designs into two constituents: time spent per layer and number of layers.

Original: Figure E in the appendix shows this breakdown for various architectures.
Revised: Figure E in the appendix shows this breakdown for various architectures.

Original: We observed that the total number of layers in a model (for ensembles this is the sum of all networks’ depth) majorly determines the training time per epoch.
Revised: We observed that the total number of layers in a model (for ensembles this is the sum of all networks’ depth) majorly determines the training time per epoch.

Original: For the same parameter budget, depthequivalent ensembles have proportionally more layers, whereas width-equivalent ensembles have proportionally more width.
Revised: For the same parameter budget, depthequivalent ensembles have proportionally more layers, whereas width-equivalent ensembles have proportionally more width.

Original: The average time per layer depends on the width and does not increase signiﬁcantly as we move from depth-equivalent ensembles to width-equivalent ensembles.
Revised: The average time per layer depends on the width and does not increase signiﬁcantly as we move from depth-equivalent ensembles to width-equivalent ensembles.

Original: On the other hand, the total number of layers scale linearly with depth.
Revised: On the other hand, the total number of layers scale linearly with depth.

Original: For the same parameter budget, total number of layers are signiﬁcantly higher for depth-equivalent ensembles as compared to the other two designs, resulting in higher per epoch training.
Revised: For the same parameter budget, total number of layers are signiﬁcantly higher for depth-equivalent ensembles as compared to the other two designs, resulting in higher per epoch training.

Original: From a GPU computing perspective, for the same parameter budget, wider and shallower networks are more efﬁcient to execute than narrower and deeper networks.
Revised: From a GPU computing perspective, for the same parameter budget, wider and shallower networks are more efﬁcient to execute than narrower and deeper networks.

Original: This can be attributed to the massive amount of data parallelism in modern GPUs.
Revised: This can be attributed to the massive amount of data parallelism in modern GPUs.

Original: Increasing the width of the network just increases the width of the kernels whereas, deepening a network introduces new operations that require additional synchronization steps.
Revised: Increasing the width of the network just increases the width of the kernels whereas, deepening a network introduces new operations that require additional synchronization steps.

Original: Networks in ensemble models converge faster than single network models for the same parameter budget.
Revised: Networks in ensemble models converge faster than single network models for the same parameter budget.

Original: The fact that ensemble designs can reach the same accuracy faster than single network models can be attributed to the fact that, for the same parameter budget, all networks in the ensemble model are smaller than the single network model.
Revised: The fact that ensemble designs can reach the same accuracy faster than single network models can be attributed to the fact that, for the same parameter budget, all networks in the ensemble model are smaller than the single network model.

Original: Smaller networks are known to converge faster albeit to a lower accuracy than larger networks.
Revised: Smaller networks are known to converge faster albeit to a lower accuracy than larger networks.

Original: However, we observe that the distinct advantage ensemble designs provide over the single model is that when we use smaller networks in an ensemble, we get the best of both worlds.
Revised: However, we observe that the distinct advantage ensemble designs provide over the single model is that when we use smaller networks in an ensemble, we get the best of both worlds.

Original: We converge faster at an individual network level and ensembling makes up for the generalization accuracy.[Figure] [Table] [Figure]
Revised: We converge faster at an individual network level and ensembling makes up for the generalization accuracy.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] Overall, our observations, again question the conventional wisdom of ensembles being signiﬁcantly slower to train as compared to single network models.
Revised: Overall, our observations, again question the conventional wisdom of ensembles being signiﬁcantly slower to train as compared to single network models.

Original: When we analyze the design space under a ﬁxed parameter budget, we uncover that for a vast range of the design space: (i) width-equivalent ensembles introduce negligible overhead to per epoch training time as compared to single network models and (ii) both ensemble designs achieve and surpass accuracy of single network models in considerably less training time.
Revised: When we analyze the design space under a ﬁxed parameter budget, we uncover that for a vast range of the design space: (i) width-equivalent ensembles introduce negligible overhead to per epoch training time as compared to single network models and (ii) both ensemble designs achieve and surpass accuracy of single network models in considerably less training time.

Original: Width-equivalent ensembles provide competitive inference time.
Revised: Width-equivalent ensembles provide competitive inference time.

Original: We provide the inference time per image in Figure 6 and observe a similar trend to training time per epoch.
Revised: We provide the inference time per image in Figure 7 and observe a similar trend to training time per epoch.

Original: While depth-equivalent ensembles are signiﬁcantly slower, width-equivalent ensembles provide comparable inference speed to single network models.
Revised: While depth-equivalent ensembles are signiﬁcantly slower, width-equivalent ensembles provide comparable inference speed to single network models.

Original: Again, this questions conventional wisdom that expects ensembles to be substantially slower in inference.
Revised: Again, this questions conventional wisdom that expects ensembles to be substantially slower in inference.

Original:  E NSEMBLES ARE M EMORY -E FFICIENT
Revised:  E NSEMBLES ARE M EMORY -E FFICIENT

Original: When we look at the metric of memory usage, we observe that the trend favors both classes of ensemble designs over single network models.
Revised: When we look at the metric of memory usage, we observe that the trend favors both classes of ensemble designs over single network models.

Original: Figure 7 provides the amount of memory used as we train depth-equivalent ensembles, width-equivalent ensembles, and single network models.
Revised: Figure 8 provides the amount of memory used as we train depth-equivalent ensembles, width-equivalent ensembles, and single network models.

Original: This is the minimum amount of memory that a GPU needs to have to train any of these designs for the batch sizes provided in Table A.
Revised: This is the minimum amount of memory that a GPU needs to have to train any of these designs for the batch sizes provided in Table A.

Original: This memory is majorly used to store model parameters and intermediate results (Jain et al., 2020).
Revised: This memory is majorly used to store model parameters and intermediate results (Jain et al., 2020).

Original: The superior memory-efﬁciency of ensemble models is due to the fact that when we train an ensemble of k networks, at any point in time, we only need as much memory to train only one of the k networks.
Revised: The superior memory-efﬁciency of ensemble models is due to the fact that when we train an ensemble of k networks, at any point in time, we only need as much memory to train only one of the k networks.

Original: This cost when compared with single networks with the same number of parameters is signiﬁcantly less.
Revised: This cost when compared with single networks with the same number of parameters is signiﬁcantly less.

Original: This memory-efﬁciency has two important implications: First, for the same GPU, we can use higher batch sizes while training an ensemble of networks.
Revised: This memory-efﬁciency has two important implications: First, for the same GPU, we can use higher batch sizes while training an ensemble of networks.

Original: This, for instance, is useful when training complex data sets such as ImageNet (Smith et al., 2018).
Revised: This, for instance, is useful when training complex data sets such as ImageNet (Smith et al., 2018).

Original: Additionally, we can feasibly train the same number of parameters in an ensemble using GPUs with less memory.
Revised: Additionally, we can feasibly train the same number of parameters in an ensemble using GPUs with less memory.

Original: In the world of cloud computing, this translates to drastic reduction in costs.
Revised: In the world of cloud computing, this translates to drastic reduction in costs.

Original: Additional results.
Revised: Additional results.

Original: We show that the same resource-related trends hold for the rest of architectures and data sets from Table A. Figure D, Figure F, Figure G, and Figure H demonstrate these trends for metrics of time to accuracy, time per epoch, inference time, and memory usage respectively.
Revised: We show that the same resource-related trends hold for the rest of architectures and data sets from Table A. Figure D, Figure F, Figure G, and Figure H demonstrate these trends for metrics of time to accuracy, time per epoch, inference time, and memory usage respectively.

Original: We also provide results on the SVHN data set in Figure I in the appendix.
Revised: We also provide results on a downsampled version of ImageNet-1K data set, CIFAR-100 with data augmentation, and SVHN data sets in Figure I, Figure J, and Figure K in the appendix respectively.

Original: Table B through Table D provide the conversion between the width factor and depth of single network models and the two ensemble design classes.
Revised: Table B through Table D provide the conversion between the width factor and depth of single network models and the two ensemble design classes.

Original:  T HE C OMPLETE P ICTURE
Revised:  T HE C OMPLETE P ICTURE

Original: The results in this paper question conventional wisdom on convolutional model design with respect to the design decision on whether to use an ensemble of networks or not.
Revised: The results in this paper question conventional wisdom on convolutional model design with respect to the design decision on whether to use an ensemble of networks or not.

Original: By creating a detailed framework that (a) allows to ﬁx resources, and (b) spans a large design space, we show that for a considerable part of the design space, given an amount of resources, ensembles (i) achieve better accuracy than single network models, (ii) train faster, (iii) provide comparable inference, and (iv) need much less memory.
Revised: By creating a detailed framework that (a) allows to ﬁx resources, and (b) spans a large design space, we show that for a considerable part of the design space, given an amount of resources, ensembles (i) achieve better accuracy than single network models, (ii) train faster, (iii) provide comparable inference, and (iv) need much less memory.

Original: Future work includes the addition of fast ensemble training approaches (such as SnapShot and MotherNets (Huang et al., 2017; Wasay et al., 2020)) and to also consider parallel training, both of which can move EST further in favor of ensembles. 
Revised: Future work includes the addition of fast ensemble training approaches (such as SnapShot, TreeNets, and MotherNets (Lee et al., 2015; Huang et al., 2017; Wasay et al., 2020)) and to also consider parallel training, both of which can move EST further in favor of ensembles. Limitations and future directions.

Revised: The analysis framework as presented in this paper covers extensively homogeneous ensembles.

Revised: There are a number of opportunities for parts of the design space not covered by this work where it would be extremely useful to verify the results further.

Revised: In particular, this includes 1) heterogenous ensembles (e.g., various ratios of width-equivalent and depthequivalent networks), 2) arbitrary network architectures (including using network architecture search approaches), and 3) application domains other than image classiﬁcation such as object detection, machine translation, and deep generative models.

Original: APPENDIX
Revised: APPENDIX

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: A I NSTABILITY OF W IDTH -E QUIVALENT E NSEMBLES

Revised: The performance of width equivalent ensembles (weq ensembles) exhibits unstable behavior.

Revised: In particular, it has local spikes when it comes to the test error rates.

Original: t e s t
Revised: This is particularly pronounced for the ResNet ensembles (Figure A(a) and Figure A(b)). Our interpretation of this phenomenon is that these local spikes have to do with the relative depth of networks in the weq ensemble designs. When comparing designs that are close together in the parameter range, we observe that the designs with more depth (of ensemble networks) generally outperform those with less depth even if the latter have more parameters. The depth of networks in the weq ensemble plays a more dominant role than the total number of parameters they have.

Revised: As an example of this, consider Figure A(a) and A(b) in the revised version of the paper (ResNet

Revised: CIFAR-10 (k=4) and ResNet CIFAR-100 (k=4).

Revised: Weq exhibit three spikes at 2.24M, 3.28M, and 5.03M parameters.

Revised: All three of these points are ﬂanked on both sides by designs that have similar number of parameters but more depth.

Revised: This observation is consistent with past observations that depth is more inﬂuential in determining the accuracy of networks (Eigen et al., 2013).

Revised: B A DDITIONAL E XPERIMENTS IN U NDERFITTING S CENARIOS

Revised: We now provide experiments on two additional data sets from underﬁtting scenarios: (i) ImageNet1K-32 a downsampled version of ImageNet-1k with 1.2 million images and 1000 labels (every image downsampled to be of size 32 × 32), and (ii) CIFAR-100 with aggressive rand-augment (Cubuk et al., 2020).

Revised: The result of these experiment can be seen in Figure I and Figure J respectively.

Revised: Overall, our observations hold for these data sets as well.

Original: e rr o r r a t
Revised: After a certain parameter budget (deﬁned in terms of [Figure] we increase the number of networks in the ensemble. Here, we demonstrate this phenomenon for ResNet models

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: ti m e t

Original: o
Revised: o

Original: [Figure] s 
Revised: s i ng l e

Original: [Figure] [Figure] [Figure] [Figure]
Revised: acc

Original:  ( m i n. )
Revised: u r ac y ( m i n

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e

Original: [Figure]
Revised: [Figure]

Original: e r l a y e r
Revised: [Table]

Original: ( m i n . )
Revised: [Figure]

Original: t
Revised: [Figure] [Figure] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: t a l l a y e r s
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: -equivalent ensembles (weq) take comparable time
Revised: t r a i n i ng ti m e p e r e po c h ( s )

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Ensemble models take longer to perform inference
Revised: i n f e r e n ce ti m e p e r i m a g e

Original: [Figure]
Revised: [Figure]

Revised: m

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t r a i n i ng ti m e p e r  e  [Figure]
Revised: m a x . m e m o r y u s a g e ( G B ) ( [Figure]

Revised: [Figure]

Revised: on CIFAR-

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original:  [Table]
Revised: [Figure] [Figure] [Figure] [Table]

Revised: Deq

Original: [Table]
Revised: [Table]

Revised: factor

Original: [Table]
Revised: [Table]

Revised: Weq

Original: [Table]
Revised: [Table]

Revised: Deq

Original: [Table]
Revised: [Table]

Revised: factor

Original: [Table]
Revised: [Table]