Original: C OUPLED E NSEMBLES OF N EURAL N ETWORKS
Revised: C OUPLED E NSEMBLES OF N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We investigate in this paper the architecture of deep convolutional networks.
Revised: We investigate in this paper the architecture of deep convolutional networks.

Original: Building on existing state of the art models, we propose a reconﬁguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN.
Revised: Building on existing state of the art models, we propose a reconﬁguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN.

Original: We show that this arrangement is an efﬁcient way to signiﬁcantly reduce the number of parameters while at the same time improving the performance.
Revised: We show that this arrangement is an efﬁcient way to signiﬁcantly reduce the number of parameters while at the same time improving the performance.

Original: The use of branches brings an additional form of regularization.
Revised: The use of branches brings an additional form of regularization.

Original: In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities.
Revised: In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities.

Original: The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently.
Revised: The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently.

Original: We refer to this branched architecture as “coupled ensembles”.
Revised: We refer to this branched architecture as “coupled ensembles”.

Original: The approach is generic and can be applied to almost any neural network architecture.
Revised: The approach is generic and can be applied to almost any neural network architecture.

Original: With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% on CIFAR-10, CIFAR-100 and
Revised: With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% on CIFAR-10, CIFAR-100 and

Original: SVHN respectively.
Revised: SVHN respectively.

Original: For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively.
Revised: For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively.

Original: With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.
Revised: With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The design of early convolutional architectures (CNN) involved choices of hyper-parameters such as: ﬁlter size, number of ﬁlters at each layer, and padding (LeCun et al., 1998; Krizhevsky et al., 2012).
Revised: The design of early convolutional architectures (CNN) involved choices of hyper-parameters such as: ﬁlter size, number of ﬁlters at each layer, and padding (LeCun et al., 1998; Krizhevsky et al., 2012).

Original: Since the introduction of the VGGNet (Simonyan & Zisserman, 2014) the design has moved towards following a template: ﬁxed ﬁlter size of 3 × 3 and N features maps, down-sample to half the input resolution only by the use of either maxpool or strided convolutions (Springenberg et al., 2015), and double the number the computed feature maps following each down-sampling operation.
Revised: Since the introduction of the VGGNet (Simonyan & Zisserman, 2014) the design has moved towards following a template: ﬁxed ﬁlter size of 3 × 3 and N features maps, down-sample to half the input resolution only by the use of either maxpool or strided convolutions (Springenberg et al., 2015), and double the number the computed feature maps following each down-sampling operation.

Original: This philosophy is used by state of the art models like ResNet (He et al., 2016b) and
Revised: This philosophy is used by state of the art models like ResNet (He et al., 2016b) and

Original: DenseNet (Huang et al., 2017b).
Revised: DenseNet (Huang et al., 2017b).

Original: The last two architectures extended the template to include the use of “skip-connections” between non-contiguous layers.
Revised: The last two architectures extended the template to include the use of “skip-connections” between non-contiguous layers.

Original: Our work extends this template by adding another element, which we refer to as “coupled ensembling”.
Revised: Our work extends this template by adding another element, which we refer to as “coupled ensembling”.

Original: In this set-up, the network is decomposed into several branches, each branch being functionally similar to a complete CNN.
Revised: In this set-up, the network is decomposed into several branches, each branch being functionally similar to a complete CNN.

Original: The proposed template achieves performance comparable to state of the art models with a signiﬁcantly lower parameter count.
Revised: The proposed template achieves performance comparable to state of the art models with a signiﬁcantly lower parameter count.

Original: In this paper, we make the following contributions: (i) we show that given a parameter budget, it is better to have the parameters split among branches rather than having a single branch (which is the case for all current networks), (ii) we compare different ways to combine the activations of the parallel branches and ﬁnd that it is best to take an arithmetic mean of the individual logprobabilities (iii) combining these elements, we signiﬁcantly match and improve the performance of convolutional networks on CIFAR and SVHN datasets, with a heavily reduced parameter count.
Revised: In this paper, we make the following contributions: (i) we show that given a parameter budget, it is better to have the parameters split among branches rather than having a single branch (which is the case for all current networks), (ii) we compare different ways to combine the activations of the parallel branches and ﬁnd that it is best to take an arithmetic mean of the individual logprobabilities (iii) combining these elements, we signiﬁcantly match and improve the performance of convolutional networks on CIFAR and SVHN datasets, with a heavily reduced parameter count.

Original: (iv) Further ensembling of coupled ensembles lead to moremore improvement.
Revised: (iv) Further ensembling of coupled ensembles lead to more improvement.

Original: This paper is organised as follows: in section 2, we discuss related work; in section 3, we introduce the concept of coupled ensembles and the motivation behind the idea; in section 4, we evaluate the proposed approach and compare it with the state of the art; and we conclude and discuss future work in section 5.
Revised: This paper is organised as follows: in section 2, we discuss related work; in section 3, we introduce the concept of coupled ensembles and the motivation behind the idea; in section 4, we evaluate the proposed approach and compare it with the state of the art; and we conclude and discuss future work in section 5.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Multi-column architectures.
Revised: Multi-column architectures.

Original: The network architecture that we propose has strong similarities with
Revised: The network architecture that we propose has strong similarities with

Original: Cires¸an’s Neural Networks Committees (Cires¸an et al., 2011) and Multi-Column Deep Neural Network (MCDNN) (Cires¸an et al., 2012), which are a type of ensemble of networks where the “committee members” or “DNN columns” correspond to our basic blocks (or branches).
Revised: Cires¸an’s Neural Networks Committees (Cires¸an et al., 2011) and Multi-Column Deep Neural Network (MCDNN) (Cires¸an et al., 2012), which are a type of ensemble of networks where the “committee members” or “DNN columns” correspond to our element blocks (or branches).

Original: However, our coupled ensemble networks differ as following: (i) we train a single model which is composed of branches, while they train each member or column separately .
Revised: However, our coupled ensemble networks differ as following: (i) we train a single model which is composed of branches, while they train each member or column separately .

Original: (ii) we have a ﬁxed parameter budget for the entire model for improving the performance.
Revised: (ii) we have a ﬁxed parameter budget for the entire model for improving the performance.

Original: This is contrary to improving it by utilising multiple models of ﬁxed size and therefore multiplying the overall size (though both are not exclusive); (iii) we combine the activations of the branches by combining their log-probabilities over the target categories, and (iv) we used the same input for all branches while they considered different preprocessing (data augmentation) blocks for different members or different subsets of columns.
Revised: This is contrary to improving it by utilising multiple models of ﬁxed size and therefore multiplying the overall size (though both are not exclusive); (iii) we combine the activations of the branches by combining their log-probabilities over the target categories, and (iv) we used the same input for all branches while they considered different preprocessing (data augmentation) blocks for different members or different subsets of columns.

Original: Multi-branch architectures.
Revised: Multi-branch architectures.

Original: Multi-branch architectures have been very successful in several vision applications (He et al., 2016a; Szegedy et al., 2015).
Revised: Multi-branch architectures have been very successful in several vision applications (He et al., 2016a; Szegedy et al., 2015).

Original: Recently, modiﬁcations have been proposed (Xie et al., 2017; Chollet, 2017) for these architectures using the concept of “grouped convolutions”, in order to factorize spatial and depth wise feature extraction.
Revised: Recently, modiﬁcations have been proposed (Xie et al., 2017; Chollet, 2017) for these architectures using the concept of “grouped convolutions”, in order to factorize spatial and depth wise feature extraction.

Original: These modiﬁcations additionally advocate the use of template building blocks stacked together to form the complete model.
Revised: These modiﬁcations additionally advocate the use of template building blocks stacked together to form the complete model.

Original: This modiﬁcation is at the level of the building blocks of their corresponding base architectures: ResNet and Inception respectively.
Revised: This modiﬁcation is at the level of the building blocks of their corresponding base architectures: ResNet and Inception respectively.

Original: In contrast we propose a generic modiﬁcation of the structure of the
Revised: In contrast we propose a generic modiﬁcation of the structure of the

Original: CNN at the global model level.
Revised: CNN at the global model level.

Original: This includes a template in which the speciﬁc architecture of a “basic block” is speciﬁed, and then this “basic block” is replicated as parallel branches to form the ﬁnal composite model.
Revised: This includes a template in which the speciﬁc architecture of a “element block” is speciﬁed, and then this “element block” is replicated as parallel branches to form the ﬁnal composite model.

Original: To further improve the performance of such architectures, Shake-Shake regularization (Gastaldi, 2017) proposes a stochastic mixture of each of the branches and has achieved good results on the
Revised: To further improve the performance of such architectures, Shake-Shake regularization (Gastaldi, 2017) proposes a stochastic mixture of each of the branches and has achieved good results on the

Original: CIFAR datasets.
Revised: CIFAR datasets.

Original: However, the number of epochs required for convergence is much higher compared to the base model.
Revised: However, the number of epochs required for convergence is much higher compared to the base model.

Original: Additionally, the technique seems to depend on the batch size.
Revised: Additionally, the technique seems to depend on the batch size.

Original: In contrast, we apply our method using the exact same hyper-parameters as used in the underlying CNN.
Revised: In contrast, we apply our method using the exact same hyper-parameters as used in the underlying CNN.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2016) investigate the usage of parallel paths in a ResNet, connecting layers across paths to allow information exchange between them.
Revised: (2016) investigate the usage of parallel paths in a ResNet, connecting layers across paths to allow information exchange between them.

Original: However this requires modiﬁcation at a local level of each of the residual blocks.
Revised: However this requires modiﬁcation at a local level of each of the residual blocks.

Original: In contrast, our method is a generic rearrangement of a given architecture’s parameters, which does not introduce additional choices.
Revised: In contrast, our method is a generic rearrangement of a given architecture’s parameters, which does not introduce additional choices.

Original: Additionally, we empirically conﬁrm that our proposed conﬁguration leads to an efﬁcient usage of parameters.
Revised: Additionally, we empirically conﬁrm that our proposed conﬁguration leads to an efﬁcient usage of parameters.

Original: Neural network ensembles.
Revised: Neural network ensembles.

Original: Ensembling is a reliable technique to increase the performance of models for a task.
Revised: Ensembling is a reliable technique to increase the performance of models for a task.

Original: Due to the presence of several local minima, multiple trainings of the exact same neural network architecture can reach a different distribution of errors on a per-class basis.
Revised: Due to the presence of several local minima, multiple trainings of the exact same neural network architecture can reach a different distribution of errors on a per-class basis.

Original: Hence, combining their outputs lead to improved performance on the overall task.
Revised: Hence, combining their outputs lead to improved performance on the overall task.

Original: This was observed very early (Hansen & Salamon, 1990) and is now commonly used for obtaining top results in classiﬁcation challenges, despite the increase in training and prediction cost.
Revised: This was observed very early (Hansen & Salamon, 1990) and is now commonly used for obtaining top results in classiﬁcation challenges, despite the increase in training and prediction cost.

Original: Our proposed model architecture is not an ensemble of independent networks given that we have a single model made up of parallel branches that is trained jointly.
Revised: Our proposed model architecture is not an ensemble of independent networks given that we have a single model made up of parallel branches that is trained jointly.

Original: This is similar in spirit to the residual block in ResNet and
Revised: This is similar in spirit to the residual block in ResNet and

Original: ResNeXt, and to the inception module in Inception but is done at the global network level.
Revised: ResNeXt, and to the inception module in Inception but is done at the global network level.

Original: We would like to emphasize here that “arranging” a given budget of parameters into parallel branches leads to an increase in performance (Tables 1, 2, 3).
Revised: We would like to emphasize here that “arranging” a given budget of parameters into parallel branches leads to an increase in performance (Tables 1, 2, 3).

Original: Additionally, the classical ensembling approach can still be applied for the fusion of independently trained coupled ensemble models, where it leads to a signiﬁcant performance improvement (Table 4).
Revised: Additionally, the classical ensembling approach can still be applied for the fusion of independently trained coupled ensemble models, where it leads to a signiﬁcant performance improvement (Table 4).

Original: Snapshot ensembles.
Revised: Snapshot ensembles.

Original: Huang et al.
Revised: Huang et al.

Original: (2017a)
Revised: (2017a)

Original: and Loshchilov & Hutter (2017) used the ensembling approach on checkpoints during the training process instead of using fully converged models.
Revised: and Loshchilov & Hutter (2017) used the ensembling approach on checkpoints during the training process instead of using fully converged models.

Original: This approach is quite efﬁcient since the obtained performance is higher for a same training time budget.
Revised: This approach is quite efﬁcient since the obtained performance is higher for a same training time budget.

Original: However, both the overall model size and prediction time are signiﬁcantly increased.
Revised: However, both the overall model size and prediction time are signiﬁcantly increased.

Original: Given a model size and performance measure, our approach aims to either keep the model size constant and improves the performance, or to obtain the same performance with a smaller model size.
Revised: Given a model size and performance measure, our approach aims to either keep the model size constant and improves the performance, or to obtain the same performance with a smaller model size.

Original:  C OUPLED ENSEMBLES T ERMINOLOGY
Revised:  C OUPLED ENSEMBLES T ERMINOLOGY

Original: For the following discussion, we deﬁne some terms:
Revised: For the following discussion, we deﬁne some terms:

Original: • Branch: the proposed model comprises several branches.
Revised: • Branch: the proposed model comprises several branches.

Original: The number of branches is denoted by e Each branch takes as input a data point and produces a score vector corresponding to the target classes.
Revised: The number of branches is denoted by e Each branch takes as input a data point and produces a score vector corresponding to the target classes.

Original: Current design of CNNs are referred to as single-branch ( e = 1 ).
Revised: Current design of CNNs are referred to as single-branch ( e = 1 ).

Original: • Basic block: the model architecture used to form a branch.
Revised: • Element block: the model architecture used to form a branch.

Original: In our experiments, we use DenseNetBC and ResNet with pre-activation as basic blocks.
Revised: In our experiments, we use DenseNet-BC and ResNet with pre-activation as element blocks.

Original: • Fuse Layer: the operation used to combine each of the parallel branches which make up our model.
Revised: • Fuse Layer: the operation used to combine each of the parallel branches which make up our model.

Original: In our experiments, each of the branches are combined by taking the average of each of their individual log probabilities over the target classes.
Revised: In our experiments, each of the branches are combined by taking the average of each of their individual log probabilities over the target classes.

Original: Section 4.4 explores different choices of operations for the fuse layer.
Revised: Section 4.4 explores different choices of operations for the fuse layer.

Original: We consider a classiﬁcation task in which individual samples are always associated to exactly one class, labelled from a ﬁnite set.
Revised: We consider a classiﬁcation task in which individual samples are always associated to exactly one class, labelled from a ﬁnite set.

Original: This is the case for CIFAR (Krizhevsky & Hinton, 2009),
Revised: This is the case for CIFAR (Krizhevsky & Hinton, 2009),

Original: SVHN (Netzer et al., 2011) and ILSVRC (Russakovsky et al., 2015) tasks.
Revised: SVHN (Netzer et al., 2011) and ILSVRC (Russakovsky et al., 2015) tasks.

Original: In theory, this generalises to other tasks as well (for example, segmentation, object detection, etc.).
Revised: In theory, this generalises to other tasks as well (for example, segmentation, object detection, etc.).

Original: We consider neural network models which output a score vector of the same dimension as the number of target classes.
Revised: We consider neural network models which output a score vector of the same dimension as the number of target classes.

Original: This is usually implemented as a linear layer and referred to as a fully connected (FC) layer.
Revised: This is usually implemented as a linear layer and referred to as a fully connected (FC) layer.

Original: This layer can be followed by a SoftMax (SM) layer to produce a probability distribution over the target classes.
Revised: This layer can be followed by a SoftMax (SM) layer to produce a probability distribution over the target classes.

Original: During training, this is followed by a loss layer, for example, negative log-likelihood (LL).
Revised: During training, this is followed by a loss layer, for example, negative log-likelihood (LL).

Original: This is the case for all recent network architectures for image
Revised: This is the case for all recent network architectures for image

Original: classiﬁcation(Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;
Revised: classiﬁcation(Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;

Original: He et al., 2016a; Xie et al., 2017; Huang et al., 2016; Zagoruyko & Komodakis, 2016; Huang et al., 2017b).
Revised: He et al., 2016a; Xie et al., 2017; Huang et al., 2016; Zagoruyko & Komodakis, 2016; Huang et al., 2017b).

Original: The differences among them is related to what is present before the last FC layer.
Revised: The differences among them is related to what is present before the last FC layer.

Original: We are agnostic to this internal setup (however complex it may or may not be) because the resulting “basic block” always takes an image as input and produces a vector of N values as output, parametrized by a tensor W .
Revised: We are agnostic to this internal setup (however complex it may or may not be) because the resulting “element block” always takes an image as input and produces a vector of N values as output, parametrized by a tensor W .

Original: In the case of ensemble of independently trained models, fusion is usually done by computing the individual predictions separately for e model instances and then averaging them.
Revised: In the case of ensemble of independently trained models, fusion is usually done by computing the individual predictions separately for e model instances and then averaging them.

Original: Each of the instances are trained seperately .
Revised: Each of the instances are trained seperately .

Original: This is functionally equivalent to predicting with a “super-network” including the e instances as parallel branches with an averaging (AVG) layer on top.
Revised: This is functionally equivalent to predicting with a “super-network” including the e instances as parallel branches with an averaging (AVG) layer on top.

Original: Such supernetworks are generally not implemented because the branch instances often already correspond to the maximum memory capacity of GPUs.
Revised: Such supernetworks are generally not implemented because the branch instances often already correspond to the maximum memory capacity of GPUs.

Original: The remaining AVG layer operation can be implemented separately.
Revised: The remaining AVG layer operation can be implemented separately.

Original: Alternatively, it is possible to place the averaging layer just after the last FC layer of the basic block instances and before the SM layer, which is then “factorized”.
Revised: Alternatively, it is possible to place the averaging layer just after the last FC layer of the element block instances and before the SM layer, which is then “factorized”.

Original: In our set-up, the model is composed of parallel branches and each branch produces a score vector for the target categories.
Revised: In our set-up, the model is composed of parallel branches and each branch produces a score vector for the target categories.

Original: The score vectors are fused through the “fuse layer” during training and the composite model produces a single prediction.
Revised: The score vectors are fused through the “fuse layer” during training and the composite model produces a single prediction.

Original: We refer to this as coupled ensembles (ﬁg. 3).
Revised: We refer to this as coupled ensembles (ﬁg. 1). [Figure]

Original: We explore three options to combine score vectors during training and for inference (section 4.4):
Revised: We explore three options to combine score vectors during training and for inference (section 4.4):

Original: • Activation (FC) average: Average the output of the FC layers of the branches • Probability (LSM) average:
Revised: • Activation (FC) average: Average the output of the FC layers of the branches • Probability (LSM) average:

Original: Average the log-probabilities that each branch predicts for the target classes.
Revised: Average the log-probabilities that each branch predicts for the target classes.

Original: • Log Likelihood (LL) average: Average the loss of each branch.
Revised: • Log Likelihood (LL) average: Average the loss of each branch.

Original: Note that for inference, averaging the FC layer activations is equivalent to averaging the logprobabilities (see section C of supplementary material).
Revised: Note that for inference, averaging the FC layer activations is equivalent to averaging the logprobabilities (see section C of supplementary material).

Original: This transformation of having multiple branches, and combining the branch score vectors by averaging the log probabilities of the target categories, leads to a performance improvement, with a lower parameter count, in all our experiments (see section 4).
Revised: This transformation of having multiple branches, and combining the branch score vectors by averaging the log probabilities of the target categories, leads to a performance improvement, with a lower parameter count, in all our experiments (see section 4).

Original: The parameter vector W of this composite branched model is the concatenation of the parameter vectors W e of the e basic blocks with 1 ≤ i ≤ e .
Revised: The parameter vector W of this composite branched model is the concatenation of the parameter vectors W e of the e element blocks with 1 ≤ i ≤ e .

Original: All parameters are in the “basic blocks” as the “fuse layer” does not contain any parameters.
Revised: All parameters are in the “element blocks” as the “fuse layer” does not contain any parameters.

Original:  E XPERIMENTS 4. D ATASETS
Revised:  E XPERIMENTS 4. D ATASETS

Original: We evaluate our proposed architecture on the CIFAR (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011) data sets.
Revised: We evaluate our proposed architecture on the CIFAR (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011) data sets.

Original: CIFAR-10 and CIFAR-100 consist of 50,000 training images and 10, test images, categorised into 10 and 100 categories respectively.
Revised: CIFAR-10 and CIFAR-100 consist of 50,000 training images and 10, test images, categorised into 10 and 100 categories respectively.

Original: SVHN consists of 73,257 training images, 531,131 “easy” training images (we use both for training) and 26,032 testing images, categorised into 10 categories.
Revised: SVHN consists of 73,257 training images, 531,131 “easy” training images (we use both for training) and 26,032 testing images, categorised into 10 categories.

Original: All images have a size of 32 × 32 pixels.
Revised: All images have a size of 32 × 32 pixels.

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: All hyper parameters are set according to the original descriptions of the “basic block” that is used.
Revised: All hyper parameters are set according to the original descriptions of the “element block” that is used.

Original: This may not be the optimal setting in our case (especially the learning rate decay schedule) but we do not alter them, so as to not introduce any bias in comparisons.
Revised: This may not be the optimal setting in our case (especially the learning rate decay schedule) but we do not alter them, so as to not introduce any bias in comparisons.

Original: For CIFAR-10, CIFAR-100 and SVHN, the input image is normalised by subtracting by the mean image and dividing by the standard deviation.
Revised: For CIFAR-10, CIFAR-100 and SVHN, the input image is normalised by subtracting by the mean image and dividing by the standard deviation.

Original: During training on CIFAR datasets, standard data augmentation is used, which comprises random horizontal ﬂips and random crops.
Revised: During training on CIFAR datasets, standard data augmentation is used, which comprises random horizontal ﬂips and random crops.

Original: For SVHN, no data augmentation is used.
Revised: For SVHN, no data augmentation is used.

Original: However, a dropout ratio of 0.2 is applied in the case of DenseNet when training on SVHN.
Revised: However, a dropout ratio of 0.2 is applied in the case of DenseNet when training on SVHN.

Original: Testing is done after normalising the input in the same way as during training.
Revised: Testing is done after normalising the input in the same way as during training.

Original: All error rates are given in percentages and correspond to an average of the last 10 epochs.
Revised: All error rates are given in percentages and correspond to an average of the last 10 epochs.

Original: This measure is more conservative than the one used by the DenseNet authors (see supplementary material, section F).
Revised: This measure is more conservative than the one used by the DenseNet authors (see supplementary material, section F).

Original: For DenseNet-BC, Amos & Kolter (2017)’s
Revised: For DenseNet-BC, Amos & Kolter (2017)’s

Original: PyTorch implementation has been used.
Revised: PyTorch implementation has been used.

Original: All execution times were measured using a single NVIDIA 1080Ti GPU with the optimal micro-batch 2 .
Revised: All execution times were measured using a single NVIDIA 1080Ti GPU with the optimal micro-batch 2 .

Original: Experiments in section 4.3, 4.4 are done on the CIFAR-100 data set with the “basic block” being
Revised: Experiments in section 4.3, 4.4 are done on the CIFAR-100 data set with the “element block” being

Original: DenseNet-BC, depth L = 100 , growth rate k = 12 .
Revised: DenseNet-BC, depth L = 100 , growth rate k = 12 .

Original: For experiments in Section 4.5, we consider this same conﬁguration (with a single branch) as our baseline reference point.
Revised: For experiments in Section 4.5, we consider this same conﬁguration (with a single branch) as our baseline reference point.

Original: 4. C OMPARISON WITH SINGLE BRANCH AND INDEPENDENT ENSEMBLES
Revised: 4. C OMPARISON WITH SINGLE BRANCH AND INDEPENDENT ENSEMBLES

Original: A natural point of comparison of the proposed branched architecture is with an ensemble of independent models.
Revised: A natural point of comparison of the proposed branched architecture is with an ensemble of independent models.

Original: Rows 2 (coupled ensemble with LSM averaging) and 4 (independent training) in
Revised: Rows 2 (coupled ensemble with LSM averaging) and 4 (independent training) in

Original: Table 1 present the results of these two cases respectively.
Revised: Table 1 present the results of these two cases respectively.

Original: Row 4 shows the error rate obtained from averaging the predictions of 4 identical models, each trained separately.
Revised: Row 4 shows the error rate obtained from averaging the predictions of 4 identical models, each trained separately.

Original: We see that even though the total number of trainable parameters involved is exactly the same, a jointly trained branched conﬁguration gives a much lower test error ( 17 . 61 vs. 18 . 42 ).
Revised: We see that even though the total number of trainable parameters involved is exactly the same, a jointly trained branched conﬁguration gives a much lower test error ( 17 . 61 vs. 18 . 42 ).

Original: The next point of comparison is with a single branch model comprising a similar number of parameters as the multi branch conﬁguration.
Revised: The next point of comparison is with a single branch model comprising a similar number of parameters as the multi branch conﬁguration.

Original: The choice of single branch models has been done by:  increasing k while keeping L constant, by increasing both k and L , or by increasing L while keeping k constant.
Revised: The choice of single branch models has been done by: 2 See section B increasing k while keeping L constant, by increasing both k and L , or by increasing L while keeping k constant.

Original: The last three rows of Table 1 show that the error from the multi branch model is considerably lower, as compared to a single branch model ( 17 . 61 vs. 20 . 01 ).
Revised: The last three rows of Table 1 show that the error from the multi branch model is considerably lower, as compared to a single branch model ( 17 . 61 vs. 20 . 01 ).

Original: This shows that the effect of the branched conﬁguration is more pronounced as the number of parameters increase (Section D. has additional results).
Revised: This shows that the effect of the branched conﬁguration is more pronounced as the number of parameters increase (Section D. has additional results).

Original: These observations show that the arranging a given budget of parameters into parallel branches is more efﬁcient in terms of parameters, as compared to having a large single branch or multiple independent models.
Revised: These observations show that the arranging a given budget of parameters into parallel branches is more efﬁcient in terms of parameters, as compared to having a large single branch or multiple independent models.

Original: In Section 4.5 we analyse the relation between the number of branches and the model performance.
Revised: In Section 4.5 we analyse the relation between the number of branches and the model performance.

Original: 4. C HOICE OF F USE L AYER OPERATION
Revised: 4. C HOICE OF F USE L AYER OPERATION

Original: In this section, we compare the performance of our proposed branched model for different choices of the “fuse layer” (see section 3).
Revised: In this section, we compare the performance of our proposed branched model for different choices of the “fuse layer” (see section 3).

Original: Experiments are carried out to evaluate the best training and prediction fusion combinations.
Revised: Experiments are carried out to evaluate the best training and prediction fusion combinations.

Original: We consider a branched model with e = 4 , trained in the following conditions: training with fusion after the FC layer, after the LSM layer, or after the LL layer.
Revised: We consider a branched model with e = 4 , trained in the following conditions: training with fusion after the FC layer, after the LSM layer, or after the LL layer.

Original: [Table]
Revised: [Table]

Original: n/a 20.02 ± 0.

Original: n/a n/a 3.29M 1. n/a 20.01 ± 0. n/a n/a 3.15M 1. Table 1 shows the performance of models under different “fuse layer” operations for inference.
Revised: Table 1 shows the performance of models under different “fuse layer” operations for inference.

Original: Note that this table includes models with parameters obtained using different training methods .
Revised: Note that this table includes models with parameters obtained using different training methods .

Original: We can make the following observations:
Revised: We can make the following observations:

Original: • The branched model with e = 4 and Avg.
Revised: • The branched model with e = 4 and Avg.

Original: LSM for the “fuse layer” has almost the same performance as a DenseNet-BC ( L = 250 , k = 24) model (Huang et al., 2017b), which has abouttimes more parameters (15.3M versus 3.2M).
Revised: LSM for the “fuse layer” has almost the same performance as a DenseNet-BC ( L = 250 , k = 24) model (Huang et al., 2017b), which has abouttimes more parameters (15.3M versus 3.2M).

Original: • The average error rate of each of the “basic blocks” trained jointly in coupled ensembles with LSM fusion is signiﬁcantly lower than the error rate of the individual instances trained separately.
Revised: • The average error rate of each of the “element blocks” trained jointly in coupled ensembles with LSM fusion is signiﬁcantly lower than the error rate of the individual instances trained separately.

Original: This indicates that the coupling not only forces them to learn complementary features as a group but also to learn better representations, individually.
Revised: This indicates that the coupling not only forces them to learn complementary features as a group but also to learn better representations, individually.

Original: Averaging the log probabilities forces the network to continuously update all branches so as to be consistent with each other.
Revised: Averaging the log probabilities forces the network to continuously update all branches so as to be consistent with each other.

Original: This provides a stronger gradient signal.
Revised: This provides a stronger gradient signal.

Original: The error gradient that is back-propagated from the fuse layer is the same for all branches, and this gradient depends on the combined predictions.
Revised: The error gradient that is back-propagated from the fuse layer is the same for all branches, and this gradient depends on the combined predictions.

Original: This means that at every step all branches act complementary to the other branches’ weight updates.
Revised: This means that at every step all branches act complementary to the other branches’ weight updates.

Original: • All ensemble combinations except the Avg.
Revised: • All ensemble combinations except the Avg.

Original: FC training do signiﬁcantly better than a single branch network.
Revised: FC training do signiﬁcantly better than a single branch network.

Original: For a parameter budget of about 3.2M, the error rate of the best single branch model is 20.01.
Revised: For a parameter budget of about 3.2M, the error rate of the best single branch model is 20.01.

Original: Keeping the same parameter budget, using 4 branches reduces the error rate to 17.61 ( − 2.40).
Revised: Keeping the same parameter budget, using 4 branches reduces the error rate to 17.61 ( − 2.40).

Original: • When training with Avg.
Revised: • When training with Avg.

Original: FC, the individual branches do not perform well (in red).
Revised: FC, the individual branches do not perform well (in red).

Original: This is expected since a similar FC average may be reached with quite unrelated FC instances.
Revised: This is expected since a similar FC average may be reached with quite unrelated FC instances.

Original: The Avg.
Revised: The Avg.

Original: FC training with Avg.
Revised: FC training with Avg.

Original: SM prediction (in red) works a bit better but is still not good because the non-linearity of the SM layer distorts the FC average.
Revised: SM prediction (in red) works a bit better but is still not good because the non-linearity of the SM layer distorts the FC average.

Original: Avg.
Revised: Avg.

Original: FC training with Avg.
Revised: FC training with Avg.

Original: FC prediction works quite well though it does not yield the best performance.
Revised: FC prediction works quite well though it does not yield the best performance.

Original: • The Avg.
Revised: • The Avg.

Original: FC prediction works at least as well and often signiﬁcantly better than the Avg.
Revised: FC prediction works at least as well and often signiﬁcantly better than the Avg.

Original: SM prediction.
Revised: SM prediction.

Original: This can be explained by the fact that the SM layer normalises values to probabilities (between 0 and 1), while the FC values remain spread and transmit more information at the FC layer.
Revised: This can be explained by the fact that the SM layer normalises values to probabilities (between 0 and 1), while the FC values remain spread and transmit more information at the FC layer.

Original: All the following experiments have Avg.
Revised: All the following experiments have Avg.

Original: LSM for the training “fuse layer” in the branched models.
Revised: LSM for the training “fuse layer” in the branched models.

Original: 4. C HOICE OF THE NUMBER OF BRANCHES
Revised: 4. C HOICE OF THE NUMBER OF BRANCHES

Original: In this section, we investigate the optimal number of branches e for a given model parameter budget.
Revised: In this section, we investigate the optimal number of branches e for a given model parameter budget.

Original: We evaluate on CIFAR-100, with DenseNet-BC as the “basic block”, and parameter budget equal to 0 .
Revised: We evaluate on CIFAR-100, with DenseNet-BC as the “element block”, and parameter budget equal to 0 .

Original: 8 M (number of parameters in DenseNet-BC ( L = 100 , k = 12) ).
Revised: 8 M (number of parameters in DenseNet-BC ( L = 100 , k = 12) ).

Original: Indeed, the optimal number of instances e is likely to depend upon the network architecture, upon the parameter budget and upon the data set
Revised: Indeed, the optimal number of instances e is likely to depend upon the network architecture, upon the parameter budget and upon the data set

Original: but this gives at least one reference.
Revised: but this gives at least one reference.

Original: This was investigated again with larger models [Table] and the results are in table 3 (last four rows). 
Revised: This was investigated again with larger models, and the results are in table 3 (last four rows). applicable ( e > 1 )

Revised: LSM

Original: [Table]
Revised: [Table]

Original: Table 2 shows the performance for different conﬁgurations of branches e , depth L , and growth rate k .
Revised: Table 2 shows the performance for different conﬁgurations of branches e , depth L , and growth rate k .

Original: One difﬁculty is that DenseNet-BC parameter counts are strongly quantiﬁed according to the L and k values ( L has to be a multiple of 6 modulo 4) and , additionally, to the e value in the coupled ensemble version.
Revised: One difﬁculty is that DenseNet-BC parameter counts are strongly quantiﬁed according to the L and k values ( L has to be a multiple of 6 modulo 4) and, additionally, to the e value in the coupled ensemble version.

Original: This is even more critical in moderate size models like the 800K one targeted here.
Revised: This is even more critical in moderate size models like the 800K one targeted here.

Original: We selected model conﬁgurations with parameters just below the target to have a fair comparison.
Revised: We selected model conﬁgurations with parameters just below the target to have a fair comparison.

Original: A few models have slightly more parameters so that some interpolation can be done for possibly more accurate comparisons.
Revised: A few models have slightly more parameters so that some interpolation can be done for possibly more accurate comparisons.

Original: We can make the following observations:
Revised: We can make the following observations:

Original: • In the considered case (DenseNet-BC, CIFAR-100 and 800k parameters), the optimal number of branches is e = 3 , L = 70 , k = 9 .
Revised: • In the considered case (DenseNet-BC, CIFAR-100 and 800k parameters), the optimal number of branches is e = 3 , L = 70 , k = 9 .

Original: With this conﬁguration, the error rates decreases from 22.
Revised: With this conﬁguration, the error rates decreases from 22.

Original: for the single branch ( L = 100 , k = 12) DenseNet-BC model to 21.10 ( − 1.77).
Revised: for the single branch ( L = 100 , k = 12) DenseNet-BC model to 21.10 ( − 1.77).

Original: • Using 2 to 4 branches yields a signiﬁcant performance gain over the classical (single branch, e = 1 ) case, and even over the original performance of 22.27 reported for the ( L = 100 , k = 12) DenseNet-BC (see supplementary material, section F).
Revised: • Using 2 to 4 branches yields a signiﬁcant performance gain over the classical (single branch, e = 1 ) case, and even over the original performance of 22.27 reported for the ( L = 100 , k = 12) DenseNet-BC (see supplementary material, section F).

Original: • Using 6 or 8 branches performs signiﬁcantly worse, possibly because the basic blocks are too ‘thin’.
Revised: • Using 6 or 8 branches performs signiﬁcantly worse, possibly because the element blocks are too ‘thin’.

Original: • Model performance is robust to slight variations of L , k and e around their optimal values, showing that the coupled ensemble approach and the DenseNet-BC architecture are quite robust relative to these choices.
Revised: • Model performance is robust to slight variations of L , k and e around their optimal values, showing that the coupled ensemble approach and the DenseNet-BC architecture are quite robust relative to these choices.

Original: • The gain in performance comes at the expense of an increased training and prediction times even though the model size does not change.
Revised: • The gain in performance comes at the expense of an increased training and prediction times even though the model size does not change.

Original: This is due to the use of smaller values of k that reduces the throughput for smaller models.
Revised: This is due to the use of smaller values of k that reduces the throughput for smaller models.

Original: 4. C OMPARISON WITH STATE OF THE ART
Revised: 4. C OMPARISON WITH STATE OF THE ART

Original: We have evaluated coupled ensembles against existing models of various sizes.
Revised: We have evaluated coupled ensembles against existing models of various sizes.

Original: We used again
Revised: We used again

Original: Huang et al.
Revised: Huang et al.

Original: (2017b)’s DenseNet-BC architecture as the “basic block” since this was the current state of the art or very close to it at the time we started these experiments.
Revised: (2017b)’s DenseNet-BC architecture as the “element block” since this was the current state of the art or very close to it at the time we started these experiments.

Original: We also evaluated
Revised: We also evaluated

Original: ResNet He et al.
Revised: ResNet He et al.

Original: (2016b)’s with pre-activation as the basic block to check if the coupled ensemble approach works well with other architectures.
Revised: (2016b)’s with pre-activation as the element block to check if the coupled ensemble approach works well with other architectures.

Original: Table 3 shows the current state of the art models (see section 2 for references) and performance of coupled ensembles in the lower part.
Revised: Table 3 shows the current state of the art models (see section 2 for references) and performance of coupled ensembles in the lower part.

Original: All results presented in this table correspond to the predictions of single model.
Revised: All results presented in this table correspond to the predictions of single model.

Original: If ensembles are considered, they are always coupled as described in section and trained as a single global model.
Revised: If ensembles are considered, they are always coupled as described in sectionand trained as a single global model.

Original: A further level of ensembling involving multiple models is considered in section 4.7.
Revised: A further level of ensembling involving multiple models is considered in section 4.7.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] ., 2017b
Revised: DenseNet L = 100 k = 24 (Huang et al., 2017b)

Original: [Table]
Revised: [Table]

Original: BC L [Table] 100 k = 12 (
Revised: DenseNet-BC L = 100 k = 12 (Huang et al., 2017b)

Original: [Table]
Revised: [Table]

Original: ., 2017b
Revised: DenseNet-BC L = 250 k =

Original: [Table]
Revised: [Table]

Original: BC L 
Revised: DenseNet-BC L = 190 k =3. - 25.6M Shake-

Original: [Table]
Revised: [Table]

Original: S
Revised: S-S

Original: [Table]
Revised: [Table]

Original: S
Revised: I (Gastaldi,

Original: [Table]
Revised: [Table]

Original: I
Revised: - - 26.2M Shake

Original: [Table]
Revised: [Table]

Original: Gastaldi,

Revised: C100 Model S

Original: [Table]
Revised: [Table]

Original: [Table] I -
Revised: I15.

Original: [Table]
Revised: - 34.4M

Original: Snapshot Ensemble DenseNet-40 ( 
Revised: Snapshot Ensemble DenseNet-40 ( α

Original: [Table]
Revised: [Table]

Original: = 0 .
Revised: 6.0M

Original: [Table] DenseNet [Table] [Table]
Revised: Snapshot Ensemble DenseNet-40 ( α 0 = 0 . 2 ) 4.

Original: [Table]
Revised: 6.0M

Original: [Table] [Table]
Revised: Snapshot Ensemble DenseNet-100 ( α 0 = 0 . 2 ) 3.

Original: [Table]
Revised: - 163M

Original: -28-10 (Loshchilov & Hutter, 2017) 4.
Revised: SGDR WRN-28-10 (Loshchilov & Hutter, 2017) 4.

Original: - 36.5M
Revised: - 36.5M

Original: SGDR WRN-28-10 3 snapshots 3.
Revised: SGDR WRN-28-10 3 snapshots 3.

Original: - 110M
Revised: - 110M

Original: ResNeXt-29, 8 × 64d (Xie et al., 2017) 3.
Revised: ResNeXt-29, 8 × 64d (Xie et al., 2017) 3.

Original: - 34.4M
Revised: - 34.4M

Original: ResNeXt-29, 16 × 64d 3.
Revised: ResNeXt-29, 16 × 64d 3.

Original: - 68.1M
Revised: - 68.1M

Original: DFN-MR2 (Zhao et al., 2016) 3.
Revised: DFN-MR2 (Zhao et al., 2016) 3.

Original: 14.9M
Revised: 14.9M

Original: DFN-MR 3.
Revised: DFN-MR 3.

Original: 24.8M
Revised: 24.8M

Original: IGC-L450M2 (Zhang et al., 2017) 3.
Revised: IGC-L450M2 (Zhang et al., 2017) 3.

Original: - 19.3M
Revised: - 19.3M

Original: IGC-L32M 3.
Revised: IGC-L32M 3.

Original: 24.1M
Revised: 24.1M

Original: ResNet pre-activation L = 65 k = 64 e =5.
Revised: ResNet pre-activation L = 65 k = 64 e =5.

Original: - 1.4M
Revised: - 1.4M

Original: ResNet pre-activation L = 164 k = 64 e =4.
Revised: ResNet pre-activation L = 164 k = 64 e =4.

Original: - 3.4M
Revised: - 3.4M

Original: ResNet pre-activation L = 164 k = 64 e =3.
Revised: ResNet pre-activation L = 164 k = 64 e =3.

Original: - 6.8M
Revised: - 6.8M

Original: DenseNet-BC L = 100 k = 12 e
Revised: DenseNet-BC L = 100 k = 12 e

Original: =4.
Revised: =4.

Original: 0.8M
Revised: 0.8M

Original: DenseNet-BC L = 112 k = 

Revised: DenseNet-BC L = 112 k =

Original: e =4.
Revised: e =4.

Original: 1.7M
Revised: 1.7M

Original: DenseNet-BC L =
Revised: DenseNet-BC L =

Original: 130 k = 20 e =3.
Revised: 130 k = 20 e =3.

Original: 3.4M
Revised: 3.4M

Original: DenseNet-BC L = 160 k = 24 e =3.
Revised: DenseNet-BC L = 160 k = 24 e =3.

Original: 6.9M
Revised: 6.9M

Original: DenseNet-BC L = 166 k = 32 e =3.
Revised: DenseNet-BC L = 166 k = 32 e =3.

Original: 13.0M
Revised: 13.0M

Original: DenseNet-BC L = 190 k = 40 e =3.
Revised: DenseNet-BC L = 190 k = 40 e =3.

Original: 25.8M
Revised: 25.8M

Original: DenseNet-BC L = 82 k
Revised: DenseNet-BC L = 82 k

Original: = 8 e =4.
Revised: = 8 e =4.

Original: 0.8M
Revised: 0.8M

Original: DenseNet-BC L = 82 k
Revised: DenseNet-BC L = 82 k

Original: = 10 e =3.
Revised: = 10 e =3.

Original: 1.6M
Revised: 1.6M

Original: DenseNet-BC L = 88 k = 14 e =3.
Revised: DenseNet-BC L = 88 k = 14 e =3.

Original: 3.5M
Revised: 3.5M

Original: DenseNet-BC L = 88 k = 20 e =3.
Revised: DenseNet-BC L = 88 k = 20 e =3.

Original: 7.0M
Revised: 7.0M

Original: DenseNet-BC L = 94 k = 26 e =3.
Revised: DenseNet-BC L = 94 k = 26 e =3.

Original: 13.0M
Revised: 13.0M

Original: DenseNet-BC L = 118 k = 35 e =2.
Revised: DenseNet-BC L = 118 k = 35 e =2.

Original: 25.7M
Revised: 25.7M

Original: DenseNet-BC L = 106 k
Revised: DenseNet-BC L = 106 k

Original: = 33 e =2.
Revised: = 33 e =2.

Original: 25.1M
Revised: 25.1M

Original: DenseNet-BC L = 76 k = 35 e =2.
Revised: DenseNet-BC L = 76 k = 35 e =2.

Original: 24.6M
Revised: 24.6M

Original: DenseNet-BC L = 64 k = 35 e =3.
Revised: DenseNet-BC L = 64 k = 35 e =3.

Original: 24.9M
Revised: 24.9M

Original: Coupled ensembles with ResNet pre-act as basic block and e = 2 , 4 leads to a signiﬁcantly better performance than single branch models, which have comparable or higher number of parameters.
Revised: Coupled ensembles with ResNet pre-act as element block and e = 2 , 4 leads to a signiﬁcantly better performance than single branch models, which have comparable or higher number of parameters.

Original: For the DenseNet-BC architecture, we considered 6 different network sizes, ranging from 0.8M up to 25.6M parameters.
Revised: For the DenseNet-BC architecture, we considered 6 different network sizes, ranging from 0.8M up to 25.6M parameters.

Original: Huang et al.
Revised: Huang et al.

Original: (2017b) reports results for the two extreme cases.
Revised: (2017b) reports results for the two extreme cases.

Original: We chose these values for the depth L and growth rate k for these points and interpolated between them according to a log scale as much as possible.
Revised: We chose these values for the depth L and growth rate k for these points and interpolated between them according to a log scale as much as possible.

Original: Our experiments show that the trade-off between L and k is not critical for a given parameter budget.
Revised: Our experiments show that the trade-off between L and k is not critical for a given parameter budget.

Original: This was also the case for choosing between the number of branches e , depth L and growth rate, k for a ﬁxed parameter budget as long as e ≥ 3 (or even e ≥ 2 for small networks).
Revised: This was also the case for choosing between the number of branches e , depth L and growth rate, k for a ﬁxed parameter budget as long as e ≥ 3 (or even e ≥ 2 for small networks).

Original: For the 6 conﬁgurations, we experimented with both the single-branch (classical) and multi-branch versions of the model, with e = 4 .
Revised: For the 6 conﬁgurations, we experimented with both the single-branch (classical) and multi-branch versions of the model, with e = 4 .

Original: Additionally, for the largest model, we tried e = 3 , 6 , 8 branches.
Revised: Additionally, for the largest model, we tried e = 3 , 6 , 8 branches.

Original: For single branch DenseNet-BC, we obtained error rates higher than reported by Huang et al.
Revised: For single branch DenseNet-BC, we obtained error rates higher than reported by Huang et al.

Original: (2017b).
Revised: (2017b).

Original: From what we have checked, their Torch7 implementation and our PyTorch one are equivalent.
Revised: From what we have checked, their Torch7 implementation and our PyTorch one are equivalent.

Original: The difference may be due to the fact that we used a more conservative measure of the error rate (on the last iterations) and from statistical differences due to different initializations and/or to non-deterministic computations (see section F in supplementary material).
Revised: The difference may be due to the fact that we used a more conservative measure of the error rate (on the last iterations) and from statistical differences due to different initializations and/or to non-deterministic computations (see section F in supplementary material).

Original: Still, the coupled ensemble leads to a signiﬁcantly better performance for all network sizes, even when compared to
Revised: Still, the coupled ensemble leads to a signiﬁcantly better performance for all network sizes, even when compared to

Original: DenseNet-BC’s reported performance.
Revised: DenseNet-BC’s reported performance.

Original: Our larger models of coupled DenseNet-BCs (error rates of 2.92% on CIFAR 10, 15.68% on CIFAR 100 and 1.50% on SVHN) perform better than or are on par with all current state of the art implementations that we are aware of at the time of submission of this work.
Revised: Our larger models of coupled DenseNet-BCs (error rates of 2.92% on CIFAR 10, 15.68% on CIFAR 100 and 1.50% on SVHN) perform better than or are on par with all current state of the art implementations that we are aware of at the time of submission of this work.

Original: Only the Shake-Shake S-S-I model (Gastaldi, 2017) performs slightly better on CIFAR 10.
Revised: Only the Shake-Shake S-S-I model (Gastaldi, 2017) performs slightly better on CIFAR 10.

Original: We also compare the performance of coupled ensembles with model architectures that were ‘learnt’ in a meta learning scenario.
Revised: We also compare the performance of coupled ensembles with model architectures that were ‘learnt’ in a meta learning scenario.

Original: The results are presented in the supplementary material, section E.
Revised: The results are presented in the supplementary material, section E.

Original: 4. E NSEMBLES OF COUPLED ENSEMBLES
Revised: 4. E NSEMBLES OF COUPLED ENSEMBLES

Original: The coupled ensemble approach is limited by the size of the network that can ﬁt into GPU memory and the training time.
Revised: The coupled ensemble approach is limited by the size of the network that can ﬁt into GPU memory and the training time.

Original: With the hardware we have access to, it was not possible to go much beyond the 25M parameters.
Revised: With the hardware we have access to, it was not possible to go much beyond the 25M parameters.

Original: For going further, we resorted to the classical ensembling approach based on independent trainings.
Revised: For going further, we resorted to the classical ensembling approach based on independent trainings.

Original: An interesting question was whether we could still signiﬁcantly improve the performance since the classical approach generally plateaus after quite a small number of models and the coupled ensemble approach already include several.
Revised: An interesting question was whether we could still signiﬁcantly improve the performance since the classical approach generally plateaus after quite a small number of models and the coupled ensemble approach already include several.

Original: For instance, SGDR with snapshots (Loshchilov & Hutter, 2017) has a signiﬁcant improvement from 1 to 3 models but not much improvement from 3 to 16 models (see tables 3 and 4).
Revised: For instance, SGDR with snapshots (Loshchilov & Hutter, 2017) has a signiﬁcant improvement from 1 to 3 models but not much improvement from 3 to 16 models (see tables 3 and 4).

Original: As doing multiple times the same training is quite costly for large, we instead ensembled the four large coupled ensemble models, e = 3 , 4 , 6 , 8 .
Revised: As doing multiple times the same training is quite costly for large, we instead ensembled the four large coupled ensemble models, e = 3 , 4 , 6 , 8 .

Original: Results are shown in table 4.
Revised: Results are shown in table 4.

Original: We obtained a signiﬁcant gain by fusing two models and a quite small one from any further fusion of three or four of them.
Revised: We obtained a signiﬁcant gain by fusing two models and a quite small one from any further fusion of three or four of them.

Original: To the best of our knowledge, these ensembles of coupled ensemble networks outperform all state of the art implementations including other ensemble-based ones at the time of submission of this work. 
Revised: To the best of our knowledge, these ensembles of coupled ensemble networks outperform all state of the art implementations including other ensemble-based ones at the time of submission of this work. [Table]

Original: [Table]
Revised: [Table]

Revised: 100M

Original: 4. P ARAMETER U SAGE
Revised: 4. P ARAMETER U SAGE

Original: The lower half of table 3, the sections with e = 1 and e > 1 highlight the difference in error rates between single and multi branch models.
Revised: The lower half of table 3, the sections with e = 1 and e > 1 highlight the difference in error rates between single and multi branch models.

Original: A branched model with 13M parameters has an error rate of 16.24 for CIFAR-100.
Revised: A branched model with 13M parameters has an error rate of 16.24 for CIFAR-100.

Original: In contrast, none of the single branch models match this performance even with double the number of parameters.
Revised: In contrast, none of the single branch models match this performance even with double the number of parameters.

Original: Figure 1 compares this with other state of the art models. 
Revised: Figure 2 compares this with other state of the art models. [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The proposed approach consists in replacing a single deep convolutional network by a number of “basic blocks” which resemble standalone CNN models.
Revised: The proposed approach consists in replacing a single deep convolutional network by a number of “element blocks” which resemble standalone CNN models.

Original: The intermediate score vectors produced by each of the basics blocks are coupled via a “fuse layer”.
Revised: The intermediate score vectors produced by each of the elements blocks are coupled via a “fuse layer”.

Original: At training time, this is done by taking an arithmetic average of their log-probabilites for the targets.
Revised: At training time, this is done by taking an arithmetic average of their log-probabilities for the targets.

Original: At test time the score vectors are averaged [Figure] [Figure] [Figure]
Revised: At test time the score vectors are averaged following the output from each score vector.

Original: Parameters

Original: E rr o r R a t e

Original: CIFARCoupled Ensembles

Original: DenseNet-BC

Original: SGDR-WRN

Original: Others

Original: [Figure]

Original: [Figure]

Original: Parameters

Original: E rr o r R a t e

Original: CIFAR-

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Parameters 1 .1 .1 .1 .1 .1 .

Original: E rr o r R a t e

Original: SVHN following the output from each score vector.

Original: Both of these aspects leads to a signiﬁcant performance improvement over a single branch conﬁguration.
Revised: Both of these aspects leads to a signiﬁcant performance improvement over a single branch conﬁguration.

Original: This improvement comes at the cost of a small increase in the training and prediction times.
Revised: This improvement comes at the cost of a small increase in the training and prediction times.

Original: The proposed approach leads to the best performance for a given parameter budget as can be seen in tables 3 and 4, and in ﬁgure 1.
Revised: The proposed approach leads to the best performance for a given parameter budget as can be seen in tables 3 and 4, and in ﬁgure 2.

Original: Additionally, the individual “basic block” performance is better as compared to when they are trained independently.
Revised: Additionally, the individual “element block” performance is better as compared to when they are trained independently.

Original: The increase in training and prediction times is mostly due to the sequential processing of branches during the forward and backward passes.
Revised: The increase in training and prediction times is mostly due to the sequential processing of branches during the forward and backward passes.

Original: The smaller size of the branches makes the data parallelism on GPUs less efﬁcient.
Revised: The smaller size of the branches makes the data parallelism on GPUs less efﬁcient.

Original: This effect is not as pronounced for larger models.
Revised: This effect is not as pronounced for larger models.

Original: This could be solved in two ways.
Revised: This could be solved in two ways.

Original: First, as there is no data dependency between the branches (before the averaging layer) it is possible to extend the data parallelism to the branches, restoring the initial level of parallelism.
Revised: First, as there is no data dependency between the branches (before the averaging layer) it is possible to extend the data parallelism to the branches, restoring the initial level of parallelism.

Original: This can be done by through a parallel implementation of multiple 2D convolutions at the same time.
Revised: This can be done by through a parallel implementation of multiple 2D convolutions at the same time.

Original: Second or alternatively, when multiple GPUs are used, it is possible to spread the branches over the
Revised: Second or alternatively, when multiple GPUs are used, it is possible to spread the branches over the

Original: GPUs.
Revised: GPUs.

Original: We have currently evaluated the coupled ensemble approach only on relatively small data sets. We therefore plan to conduct experiments on ImageNet (Russakovsky et al., 2015)  to check whether it will work equally well on larger collections.
Revised: Preliminary experiments on ImageNet (Russakovsky et al., 2015) show that coupled ensembles have a lower error for the same parameter budget as compared to single branch models. We will expand on these experiments in the future.

Original: Supplementary Material A I MPLEMENTATION
Revised: Supplementary Material A I MPLEMENTATION

Original: Figure 2 shows the common structure of the test (top) and train (bottom) versions of networks used as basic blocks.
Revised: Figure 3 shows the common structure of the test (top) and train (bottom) versions of networks used as element blocks.

Original: Figure 3 shows how it is possible in the test version to place the averaging layer just after the last FC layer of the basic block instances and before the SM layer which is then “factorized”.
Revised: Figure 4 shows how it is possible in the test version to place the averaging layer just after the last FC layer of the element block instances and before the SM layer which is then “factorized”.

Original: The e model instances do not need to share the same architecture.
Revised: The e model instances do not need to share the same architecture.

Original: Figure 4 shows how it is possible in the train version to place the averaging layer after the last FC layer, after the SM (actually LSM (LogSoftMax), which is equivalent to do a geometric mean of the SM values) layer, or after the LL layer.
Revised: Figure 5 shows how it is possible in the train version to place the averaging layer after the last FC layer, after the SM (actually LSM (LogSoftMax), which is equivalent to do a geometric mean of the SM values) layer, or after the LL layer.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We reuse “basic blocks” from other groups (with appropriate credits, please let us know if any is missing or requires updating) in their original form as much as possible both for efﬁciency and for ensuring more meaningful comparisons.
Revised: We reuse “element blocks” from other groups (with appropriate credits, please let us know if any is missing or requires updating) in their original form as much as possible both for efﬁciency and for ensuring more meaningful comparisons.

Original: When available, we use the PyTorch implementations.
Revised: When available, we use the PyTorch implementations.

Original: Each of the e branches is deﬁned by a parameter vector W e containing the same parameters as the original implementation.
Revised: Each of the e branches is deﬁned by a parameter vector W e containing the same parameters as the original implementation.

Original: The global network is deﬁned by a parameter vector W which is a concatenation of all the W e parameter vectors.
Revised: The global network is deﬁned by a parameter vector W which is a concatenation of all the W e parameter vectors.

Original: When training is done in the coupled mode and the prediction is done in a separate mode or vice-versa, a dedicated script is used for splitting the W vector into the W e ones or vice-versa.
Revised: When training is done in the coupled mode and the prediction is done in a separate mode or vice-versa, a dedicated script is used for splitting the W vector into the W e ones or vice-versa.

Original: In all coupled networks, for all train versions and for all test version, the same global parameter vector W is used with the same split and deﬁning the same basic block functions.
Revised: In all coupled networks, for all train versions and for all test version, the same global parameter vector W is used with the same split and deﬁning the same element block functions.

Original: This is how we can combine in any way all of the four possible training conditions with all the three possible prediction conditions, even though not all of them are consistent or equally efﬁcient.
Revised: This is how we can combine in any way all of the four possible training conditions with all the three possible prediction conditions, even though not all of them are consistent or equally efﬁcient.

Original: The overall network architecture is determined by:
Revised: The overall network architecture is determined by:

Original: • the global hyper-parameter specifying the train versus test mode; • the global hyper-parameter e specifying the number of branches; • the global hyper-parameter specifying after which layer the AVG layer should be placed (FC, SM or LL);
Revised: • the global hyper-parameter specifying the train versus test mode; • the global hyper-parameter e specifying the number of branches; • the global hyper-parameter specifying after which layer the AVG layer should be placed (FC, SM or LL);

Original: [Figure]
Revised: [Figure]

Original: • either one basic block to be replicated e times with its own hyper-parameters or a list of e basic blocks, each with its own hyper-parameters
Revised: • either one element block to be replicated e times with its own hyper-parameters or a list of e element blocks, each with its own hyper-parameters.

Original: B M ICRO  BATCH VERSUS  BATCH
Revised: B M ICRO - BATCH VERSUS MINI - BATCH

Original: For some of the larger models, it was not possible to train them with a (mini-)batch size of 64.
Revised: For some of the larger models, it was not possible to train them with a (mini-)batch size of 64.

Original: In this case, we split data batches into m “micro-batches” with b/m elements each, b being equal to batch size.
Revised: In this case, we split data batches into m “micro-batches” with b/m elements each, b being equal to batch size.

Original: We accumulate the gradient over these micro batches and take the average over the m micro-batches to get an almost equivalent gradient, as we would have got if we processed data directly as a single batch.
Revised: We accumulate the gradient over these micro batches and take the average over the m micro-batches to get an almost equivalent gradient, as we would have got if we processed data directly as a single batch.

Original: The gradient would have been exactly equivalent but for the BatchNorm layer.
Revised: The gradient would have been exactly equivalent but for the BatchNorm layer.

Original: This is because
Revised: This is because

Original: BatchNorm uses the batch mean and standard deviation to normalize the activations during the forward pass.
Revised: BatchNorm uses the batch mean and standard deviation to normalize the activations during the forward pass.

Original: This means that the micro-batch statistics are used whereas, ideally to get an exact equivalent, we would need the whole batch statistics.
Revised: This means that the micro-batch statistics are used whereas, ideally to get an exact equivalent, we would need the whole batch statistics.

Original: However, in practice this does not make a signiﬁcant difference.
Revised: However, in practice this does not make a signiﬁcant difference.

Original: Hence, to have the same settings for comparison among different models, we perform parameter updates using gradient for a batch, while performing forward passes with micro-batches (to have an optimal throughput).
Revised: Hence, to have the same settings for comparison among different models, we perform parameter updates using gradient for a batch, while performing forward passes with micro-batches (to have an optimal throughput).

Original: C T EST TIME EQUIVALENCE BETWEEN FC AVERAGE AND L OG S OFT M AX AVERAGE
Revised: In the single branch case for a given parameter budget, the need for memory depends mostly on the network depth and on the mini-batch (or micro-batch) size. We use the micro-batch “trick” for adjusting the memory need to what is available while keeping the mini-batch size equal to the default value (64 for CIFAR and SVHN). Though this is not strictly equivalent, this does not hurt performance and this would even yield an slight increase. The multi-branch version does not require more memory if the branches’ width is kept constant.

Revised: More memory is needed only of the branches’ width is reduced.

Revised: Indeed, if we use branches with a constant parameter budget, we have to reduce either the width or the depth or both.

Revised: We did some hyper-parameter search experiments by cross-validation and these indicated that the best option was a reduction of both while the exact trade-off was not very critical.

Original: {+In practice, for our “full-size” experiments ( 25M parameters) we did the training within the 11GB memory of GTX 
+}
Revised: {+In practice, for our “full-size” experiments ( 25M parameters) we did the training within the 11GB memory of GTX 
+}

Revised: Ti using micro-batch sizes of 16 for the single-branch versions and of 8 for multi-branch ones.

Revised: Splitting the network over two GPU boards allows for doubling the micro-batch sizes.

Revised: However, this usually does not signiﬁcantly increases the speed and this does not either leads to a performance (Toperror rate) improvement.

Revised: C T EST TIME EQUIVALENCE BETWEEN FC AVERAGE AND L OG S OFT M AX AVERAGE

Original: Given branches E = E 1 , E 2 , ..E e , each E i produces a score vector of dimension C , where C is the number of categories.
Revised: Given branches E = E 1 , E 2 , ..E e , each E i produces a score vector of dimension C , where C is the number of categories.

Original: An element of E i is referenced as E ci , were c ∈ [1 , C ] .
Revised: An element of E i is referenced as E ci , were c ∈ [1 , C ] .

Original: FC Average denotes averaging the raw activations from each branch.
Revised: FC Average denotes averaging the raw activations from each branch.

Original: LSM Average denotes averaging across branches, after a LogSoftMax [Table] Case [Table]
Revised: LSM Average denotes averaging across branches, after a LogSoftMax operation in applied on each branch activation vector, separately.

Original: [Equation]
Revised: [Equation]

Original: LSM average: Scores cLSM
Revised: LSM average: Scores cLSM

Original: =
Revised: =

Original: (cid:80) ei =1 E ci − (cid:80) ei =1 Z i , where Z e = log (cid:80) C exp( E ce ) .
Revised: (cid:80) ei [Equation] 1 E ci − (cid:80) ei =1 Z i , where Z e = log (cid:80) C exp( E ce ) .

Original: Hence we see that the LSM average score vector is a translated version of the FC average score vector.
Revised: Hence we see that the LSM average score vector is a translated version of the FC average score vector.

Original: Also, doing an arithmetic average of LogSoftMax values is equivalent to doing a geometric average of
Revised: Also, doing an arithmetic average of LogSoftMax values is equivalent to doing a geometric average of

Original: SoftMax values.
Revised: SoftMax values.

Original: This holds during inference where we are interested only in the maximum value
Revised: This holds during inference where we are interested only in the maximum value.

Original: [Table]

Original: D A DDITIONAL RESULTS D. C HOICE OF FUSION LAYER
Revised: D A DDITIONAL RESULTS D. C HOICE OF FUSION LAYER

Original: Table 5 shows the same results as in table 1 but for coupled ensembles with two branches only.
Revised: Table 5 shows the same results as in table 1 but for coupled ensembles with two branches only.

Original: The observations are the same as with four branches (see section 4.4).
Revised: The observations are the same as with four branches (see section 4.4).

Original: Even using only two branches provide a signiﬁcant gain over a single branch architecture of comparable size.
Revised: Even using only two branches provide a signiﬁcant gain over a single branch architecture of comparable size.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: ± 22. 1.60M 0. SM 22.17 ± 0. 1.60M 0.

Original: n
Revised: LL 22.78 ± 0. 1.60M 0.

Original: [Table] [Table]
Revised: none 23.13 ± 0.15(*) 20.

Original: n
Revised: 1.60M 0. n/a 21.22 ± 0. n/a n/a 1.57M 0.

Original: [Table]
Revised: n/a 21.75 ± 0.

Original: [Table]
Revised: n/a n/a 1.55M 0. n/a 20.80 ± 0. n/a n/a 1.56M 0.

Original: D. N UMBER OF BRANCHES
Revised: D. N UMBER OF BRANCHES

Original: Table 6 is an extended version of table 2 in which variation of the depth L and the growth rate k are also evaluated for an approximately ﬁxed parameter count.
Revised: Table 6 is an extended version of table 2 in which variation of the depth L and the growth rate k are also evaluated for an approximately ﬁxed parameter count.

Original: The performance is quite stable against variation of the ( L, k ) compromise.
Revised: The performance is quite stable against variation of the ( L, k ) compromise.

Original: The same experiment was done on a validation set with a 40k/10k random split of the CIFAR100 training set and we could draw the same conclusions from there; they led to predict that the ( L = 82 , k = 8 , e = 3) combination should be the best one on the test set.
Revised: The same experiment was done on a validation set with a 40k/10k random split of the CIFAR100 training set and we could draw the same conclusions from there; they led to predict that the ( L = 82 , k = 8 , e = 3) combination should be the best one on the test set.

Original: The ( L = 70 , k = 9 , e = 3) combination appeared to be slightly better here but the difference is probably not statistically signiﬁcant. 
Revised: The ( L = 70 , k = 9 , e = 3) combination appeared to be slightly better here but the difference is probably not statistically signiﬁcant. [Table]

Original: E C OMPARISON WITH Learnt ARCHITECTURES
Revised: E C OMPARISON WITH Learnt ARCHITECTURES

Original: In table 7, we compare the parameter usage and performance of the branched coupled ensembles with model architectures that were recovered using meta learning techniques. 
Revised: In table 7, we compare the parameter usage and performance of the branched coupled ensembles with model architectures that were recovered using meta learning techniques. [Table]

Original: F P ERFORMANCE MEASUREMENT AND REPRODUCIBILITY ISSUES
Revised: F P ERFORMANCE MEASUREMENT AND REPRODUCIBILITY ISSUES

Original: When attempting to compare the relative performance of different methods, we face the issue of the reproducibility of the experiments and of the statistical signiﬁcance of the observed difference be- [Table]
Revised: When attempting to compare the relative performance of different methods

Original: [Table]
Revised: [Table]

Original: tween performance measures.
Revised: we face the issue of the reproducibility of the experiments and of the statistical signiﬁcance of the observed difference between performance measures.

Original: Even for a same experiment
Revised: Even for a same experiment

Original: [Table]
Revised: [Table]

Original: we identiﬁed the ﬁve following sources of variation in the performance measure:
Revised: we identiﬁed the ﬁve following sources of variation in the performance measure:

Original: • Underlying framework for the implementation: we made experiments with Torch7 (lua) and with PyTorch.
Revised: • Underlying framework for the implementation: we made experiments with Torch7 (lua) and with PyTorch.

Original: • Random seed for the network initialization.
Revised: • Random seed for the network initialization.

Original: • CuDNN non-determinism during training: GPU associative operations are by default fast but non-deterministic.
Revised: • CuDNN non-determinism during training: GPU associative operations are by default fast but non-deterministic.

Original: We observed that the results varies even for a same tool and the same seed.
Revised: We observed that the results varies even for a same tool and the same seed.

Original: In practice, the observed variation is as important as when changing the seed.
Revised: In practice, the observed variation is as important as when changing the seed.

Original: • Fluctuations associated to the computed moving average and standard deviation in batch normalization: these ﬂuctuations can be observed even when training with the learning rate, the SGD momentum and the weight decay all set to 0 .
Revised: • Fluctuations associated to the computed moving average and standard deviation in batch normalization: these ﬂuctuations can be observed even when training with the learning rate, the SGD momentum and the weight decay all set to 0 .

Original: During the last few epochs of training, their level of inﬂuence is the sameas with the default value of these hyper-parameters.
Revised: During the last few epochs of training, their level of inﬂuence is the same

Original: • Choice of the model instance chosen from training epochs: the model obtained after the last epoch, or the best performing model.
Revised: • Choice of the model instance chosen from training epochs: the model obtained after the last epoch, or the best performing model.

Original: Note that choosing the best performing model involves looking at test data.
Revised: Note that choosing the best performing model involves looking at test data.

Original: Regardless of the implementation, the numerical determinism, the Batch Norm moving average, and the epoch sampling questions, we should still expect a dispersion of the evaluation measure according to the choice of the random initialization since different random seeds will likely lead to different local minima.
Revised: Regardless of the implementation, the numerical determinism, the Batch Norm moving average, and the epoch sampling questions, we should still expect a dispersion of the evaluation measure according to the choice of the random initialization since different random seeds will likely lead to different local minima.

Original: It is generally considered that the local minima obtained with “properly designed and trained” neural networks should all have similar performance (Kawaguchi, 2016).
Revised: It is generally considered that the local minima obtained with “properly designed and trained” neural networks should all have similar performance (Kawaguchi, 2016).

Original: We do observe a relatively small dispersion (quantiﬁed by the standard deviation below) conﬁrming this hypothesis.
Revised: We do observe a relatively small dispersion (quantiﬁed by the standard deviation below) conﬁrming this hypothesis.

Original: This dispersion may be small but it is not negligible and it complicates the comparisons between methods since differences in measures lower than their dispersions is likely to be nonsigniﬁcant.
Revised: This dispersion may be small but it is not negligible and it complicates the comparisons between methods since differences in measures lower than their dispersions is likely to be nonsigniﬁcant.

Original: Classical statistical signiﬁcance tests do not help much here since differences that are statistically signiﬁcant in this sense can be observed between models obtained just with different seeds (and even with the same seed), everything else being kept equal.
Revised: Classical statistical signiﬁcance tests do not help much here since differences that are statistically signiﬁcant in this sense can be observed between models obtained just with different seeds (and even with the same seed), everything else being kept equal.

Original: Experiments reported in this section gives an estimation of the dispersion in the particular case of a moderate scale model.
Revised: Experiments reported in this section gives an estimation of the dispersion in the particular case of a moderate scale model.

Original: We generally cannot afford doing a large number of trials for larger models.
Revised: We generally cannot afford doing a large number of trials for larger models.

Original: We tried to quantify the relative importance of the different effects in the particular case of DenseNetBC with L = 100 , k = 12 on CIFAR 100.
Revised: We tried to quantify the relative importance of the different effects in the particular case of DenseNetBC with L = 100 , k = 12 on CIFAR 100.

Original: The upper part of table 8 shows the results obtained for the same experiment in the four groups of three rows.
Revised: The upper part of table 8 shows the results obtained for the same experiment in the four groups of three rows.

Original: We tried four combinations corresponding to the use of Torch7 versus PyTorch and to the use of the same seed versus the use of different seeds.
Revised: We tried four combinations corresponding to the use of Torch7 versus PyTorch and to the use of the same seed versus the use of different seeds.

Original: For each of these conﬁguration, we used as the performance measure: (i) the error rate of the model computed at the last epoch or (ii) the average of the error rate of the models computed at the lastepochs, (iii) the error rate of the model having the lowest error rate over all epochs.
Revised: For each of these conﬁguration, we used as the performance measure: (i) the error rate of the model computed at the last epoch or (ii) the average of the error rate of the models computed at the lastepochs, (iii) the error rate of the model having the lowest error rate over all epochs.

Original: For these 2 × 2 ×cases, we present the minimum, the median, the maximum and the mean ± standard deviation over 10 measures corresponding to 10 identical runs (except for the seed when indicated).
Revised: For these 2 × 2 ×cases, we present the minimum, the median, the maximum and the mean ± standard deviation over 10 measures corresponding to 10 identical runs (except for the seed when indicated).

Original: Additionally, in the case of the average of the error rate of the models computed at the 10 last epochs, we present the root mean square of the standard deviation of the ﬂuctuations on the last 10 epochs (which is the same as the square root of the mean of their variance).
Revised: Additionally, in the case of the average of the error rate of the models computed at the 10 last epochs, we present the root mean square of the standard deviation of the ﬂuctuations on the last 10 epochs (which is the same as the square root of the mean of their variance).

Original: We make the following observations:
Revised: We make the following observations:

Original: • There does not seem to be a signiﬁcant difference between Torch7 and PyTorch implementations; • There does not seem to be a signiﬁcant difference between using a same seed and using different seeds; the dispersion observed using the same seed (with everything else being equal) implies that there is no way to exactly reproduce results; • There does not seem to be a signiﬁcant difference between the means over the 10 measures computed on the single last epoch and the means over the 10 measures computed on the last 10 epochs; • The standard deviation of the measures computed on the 10 runs is slightly but consistently smaller when the measures are computed on the last 10 epochs than when they are computed on the single last epoch; this is the same for the difference between the best and the worst measures; this was expected since averaging the measure on the last 10 epochs reduces the ﬂuctuations due to the moving average and standard deviation computed in batch normalization and possibly too the the random ﬂuctuations due to the ﬁnal learning steps; • The mean of the measures computed on the 10 runs is signiﬁcantly lower when the measure is taken at the best epoch than when they are computed either on the single last epoch or on the last 10 epochs.
Revised: • There does not seem to be a signiﬁcant difference between Torch7 and PyTorch implementations; • There does not seem to be a signiﬁcant difference between using a same seed and using different seeds; the dispersion observed using the same seed (with everything else being equal) implies that there is no way to exactly reproduce results; • There does not seem to be a signiﬁcant difference between the means over the 10 measures computed on the single last epoch and the means over the 10 measures computed on the last 10 epochs; • The standard deviation of the measures computed on the 10 runs is slightly but consistently smaller when the measures are computed on the last 10 epochs than when they are computed on the single last epoch; this is the same for the difference between the best and the worst measures; this was expected since averaging the measure on the last 10 epochs reduces the ﬂuctuations due to the moving average and standard deviation computed in batch normalization and possibly too the the random ﬂuctuations due to the ﬁnal learning steps; • The mean of the measures computed on the 10 runs is signiﬁcantly lower when the measure is taken at the best epoch than when they are computed either on the single last epoch or on the last 10 epochs.

Original: This is expected since the minimum is always below the average.
Revised: This is expected since the minimum is always below the average.

Original: However, presenting this measure involves using the test data for selecting the best model.
Revised: However, presenting this measure involves using the test data for selecting the best model.

Original: Following these observations, we propose a method for ensuring the best reproducibility and the fairest comparisons.
Revised: Following these observations, we propose a method for ensuring the best reproducibility and the fairest comparisons.

Original: Choosing the measure as the minimum of the error rate for all models computed during the training seems neither realistic nor a good practice since we have no way to know which model will be the best one without looking at the results (cross-validation cannot be used for that) and this is like tuning on the test set.
Revised: Choosing the measure as the minimum of the error rate for all models computed during the training seems neither realistic nor a good practice since we have no way to know which model will be the best one without looking at the results (cross-validation cannot be used for that) and this is like tuning on the test set.

Original: Even though this is not necessarily unfair for system comparison if the measures are done in this condition for all systems, this does introduce a bias for the absolute performance estimation.
Revised: Even though this is not necessarily unfair for system comparison if the measures are done in this condition for all systems, this does introduce a bias for the absolute performance estimation.

Original: Using the error rate at the last iteration or at the 10 last iteration does not seem to make a difference in the mean but the standard deviation is smaller for the latter, therefore this one should be preferred when a single experiment is conducted.
Revised: Using the error rate at the last iteration or at the 10 last iteration does not seem to make a difference in the mean but the standard deviation is smaller for the latter, therefore this one should be preferred when a single experiment is conducted.

Original: We also checked that using the 10 or the 25 last epochs does not make much difference (learning at this point does not seem to  lead to further improvement).
Revised: We also checked that using the 10 or the 25 last epochs does not make much difference (learning at this point does not seem to [Table] lead to further improvement).

Original: A value different from 10 can be used and this is not critical.
Revised: A value different from 10 can be used and this is not critical.

Original: In all the CIFAR experiments reported in this paper, we used the average of the error rate for the models obtained at the last 10 epochs as this should be (slightly) more robust and more conservative.
Revised: In all the CIFAR experiments reported in this paper [Table] we used the average of the error rate for the models obtained at the last 10 epochs as this should be (slightly) more robust and more conservative.

Original: The case for SVHN experiments is slightly different since there is a much smaller number of much bigger epochs; we used the last 4 iterations in this case.
Revised: The case for SVHN experiments is slightly different since there is a much smaller number of much bigger epochs; we used the last 4 iterations in this case.

Original: These observations have been made in a quite speciﬁc case but the principle and the conclusions (use of the average of the error rate from the last epochs should lead to more robust and conservative results) are likely to be general.
Revised: These observations have been made in a quite speciﬁc case but the principle and the conclusions (use of the average of the error rate from the last epochs should lead to more robust and conservative results) are likely to be general.

Original: Table 8 also shows the results for a coupled ensemble network of comparable size, for a coupled ensemble network four times bigger, and for a coupled ensemble network approximately 32 times bigger.
Revised: Table 8 also shows the results for a coupled ensemble network of comparable size, for a coupled ensemble network four times bigger, and for a coupled ensemble network approximately 32 times bigger.

Original: Similar observations can be made and, additionally, we can 
Revised: Similar observations can be made and, additionally, we can observe that both the range and the standard deviations are smaller for networks of comparable sizes.

Original: [Table]
Revised: This might be because an averaging is already made between the branches leading to a reduction of the variance.

Original: n [Table]
Revised: G T RAINING TIME BUDGET

Original: observe that both the range and the standard deviations are smaller for  networks  of comparable sizes.
Revised: In this study, all comparisons between single-branch (classical) and multi-branch architectures have been made at constant parameter budget and they have shown a clear advantage for the multi-branch networks under this condition. However, the training time of multi-branch networks is currently signiﬁcantly longer than the training time of the corresponding single branch network. Though this might be improved by better parallelization, we investigate here whether the multi-branch architectures can still improve over single-branch ones at constant training time budget with the current implementation.

Original: This might be because an averaging is already made between the branches leading to  a reduction of  the variance.
Revised: There are several ways to reduce the training time: (i) reduce the number of iterations; (ii) reduce the parameter count; and/or (iii) increase the width while reducing the depth for a constant parameter count. Table 9 shows the results obtained for these three options while taking as a baseline, the single branch DenseNet-BC L = 190 , k = 40 , e = 1 , which has a training time of about 80 hours using a single NVIDIA GTX 1080Ti. Results are shown for CIFAR 10 and 100 with statistics on 5 runs for each conﬁguration. The corresponding multi-branch baseline with the same parameter budget is DenseNet-BC L = 106 , k

Revised: = 33 , e = 4 , whose training time for 300 epochs is about 1.6 times that of the baseline’s.

Revised: Option (i) correspond to reducing the number of training epochs to 188; option (ii) corresponds to reducing the depth L from 106 to 88, thereby reducing the parameter count (reducing the width increases the training time); and option (iii) matches both the parameter count and training time of the single-branch baseline with DenseNet-BC L = 76 , k = 44 , e = 4 .

Revised: Additionally, results are shown for DenseNet-BC L = 94 , k = 26 , e = 4 and for DenseNet-BC

Revised: L = 88 , k = 20 , e = 4 , which respectively correspond to a parameter budget divided by 2 and by 3.7.

Revised: [Table]

Revised: Even though they perform slightly worse than the full multi-branch baseline, all three options still perform signiﬁcantly better than the single-branch baseline (with differences between them possibly not statistically signiﬁcant).

Revised: DenseNet-BC L = 88 , k

Revised: = 20 , e = 4 still performs better than the single-branch baseline with a parameter count divided by 3.7 and a training time divided by 1.5.

Revised: H T RAINING DATA EFFICIENCY

Revised: In this section we compare the performance of single branch models and coupled ensembles in the context of a low training data scenario.

Revised: We train a single branch model and multi-branch coupled ensemble, both having the same number of parameters, on two datasets: STL-10 (Coates et al., 2011) and a 10K balanced random subset of the 50K CIFAR-100 training set.

Revised: STL-10 comprises images of size 96x96, grouped into 10 classes.

Revised: Each class has 500 training images.

Revised: The test set has 8000 images.

Revised: Results are shown in table 10.

Revised: We see that for a ﬁxed parameter budget, coupled ensembles signiﬁcantly outperform the single branch model.

Revised: [Table]

Revised: I P RELIMINARY EXPERIMENTS ON I MAGE N ET

Revised: We conducted preliminary experiments on ILSVRC2012 (Russakovsky et al., 2015) to compare single-branch model and multi-branch coupled ensembles.

Revised: Due to time, storage and memory constraints on our machines, these experiments were done on images of size 256 × 256, instead of the original sized images.

Revised: Data augmentation involved only random horizontal ﬂips, and random crops of size 224 × 224.

Revised: For a baseline single-branch model, we use DenseNet-169-k32-e1 and we compare this with coupled ensemble DenseNet-121-k30-e2.

Revised: We realize this is not a state-of-the-art baseline but due to the constraints we had this was the strongest possible.

Revised: We are carrying out experiments with full sized images and increased data augmentation.

Revised: We will update the manuscript when they become available but they will be after the deadline.

Revised: The current results are displayed in table 11 and they show that the coupled ensemble approach with two branches produces a signiﬁcant improvement over the baseline, even wit a constant training time budget.

Revised: [Table]