Original: IEA: I NNER E NSEMBLE A VERAGE WITHIN A CONVOLUTIONAL NEURAL NETWORK
Revised: IEA: I NNER E NSEMBLE A VERAGE WITHIN A CONVOLUTIONAL NEURAL NETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Ensemble learning is a method of combining multiple trained models to improvethe model accuracy.
Revised: Ensemble learning is a method of combining multiple trained models to improve model accuracy.

Original: We introduce the usage of such methods, speciﬁcally ensemble average inside Convolutional Neural Networks (CNNs) architectures. By Inner Average Ensemble (IEA) of multiple convolutional neural layers (CNLs) replacing the single CNL inside the CNN architecture, the accuracy of the CNN increased.
Revised: We propose the usage of such methods, speciﬁcally ensemble average, inside Convolutional Neural Network (CNN) architectures by replacing the single convolutional layers with Inner Average Ensembles (IEA) of multiple convolutional layers. Empirical results on different benchmarking datasets show that CNN models using IEA outperform those with regular convolutional layers and advances the state of art.

Original: A visual and a similarity score analysis of the features generated from IEA explains why it boosts the model performance.
Revised: A visual and a similarity score analysis of the features generated from IEA explains why it boosts the model performance

Original: Empirical results using different benchmarking datasets and well-known deep model architectures shows that IEA outperforms the ordinary CNL used in CNNs

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Ensemble learning Zhou (2012); Rokach (2010) is the method of combining multiple models trained over the same dataset or  random sets ofthe datasetlike the methods of bagging and boosting Opitz & Maclin (1999) to improve the model prediction. A different variation is to ensemble the same model at different training epochs Qiu et al.
Revised: Ensemble learning (Rokach, 2010; Zhou, 2012) is the method of combining multiple models trained over the same dataset or a random set of dataset to improve the model performance.

Original: The methods of ensemble have been widely used in deep learning Qiu et al. Dietterich (2000); Drucker et al. (1994) to improve the overall model accuracy.
Revised: The methods of ensemble have been widely used in deep learning (Qiu et al., 2014; Dietterich, 2000; Drucker et al., 1994) to improve the overall model accuracy.

Original: Ensembling neural networks  is known to reduce the variance in prediction, in other words it helps the network to generalize more than the usage of a one network see for instance Krogh & 
Revised: Combining neural networks in ensemble is known to reduce the variance in prediction, in other words it helps the network to generalize more than the usage of a one network (Krogh & Vedelsby, 1995;

Original: Vedelsby (1995); Geman et al. ; Zhou et al.
Revised: Geman et al., 1992; Zhou et al., 2002).

Original: The work done by Stahlberg & Byrne (2017) proposed a method of ensembling multiple models where it unfolds the ensemble into a larger network.
Revised: The work done by (Stahlberg & Byrne, 2017) proposed a method of combining multiple models where it unfolds the ensemble into a larger network.

Original: Lee et al. (2015) discussed the power of ensemble in training CNNs, they proposed a method for training ensemble by a speciﬁc loss function rather than averaging the predictions of the models.
Revised: (Lee et al., 2015) discussed the power of ensemble in training CNN and proposed a method for training ensemble by a speciﬁc loss function rather than averaging the predictions of the models.

Original: Convolutional neural networks (CNNs) Lecun et al. (1998) are extremely successful architectures that are widely used in different areas like computer vision Krizhevsky et al. (2012), text analysis dos Santos & Gatti (2014); Lai et al.
Revised: Convolutional Neural Networks (CNNs) (Lecun et al., 1998) are extremely successful architectures that are widely used in different areas such as computer vision (Krizhevsky et al., 2012), text analysis (dos Santos & Gatti, 2014; Lai et al., 2015) and even general temporal sequence problems (Bai et al., 2018).

Original: (2015) and even in general temporal sequence problems Bai et al.

Original: CNNs are a biologically inspired simulation of the cats visual cortex Hubel & Wiesel (1968).
Revised: CNN is a biologically inspired simulation of the cats visual cortex (Hubel & Wiesel, 1968).

Original: Usually CNNs are composed of a convolutionalneural layer(CNL) followed by a pooling layer.
Revised: Usually CNN is composed of a convolutional layer followed by a pooling layer.

Original: The CNL and pool layers arerepeated and stacked to introduce more depth inside the model being constructed.
Revised: Alternating convolutional layers and pooling layers are stacked to introduce more depth inside the model being constructed.

Original: We call the connection of CNLs and pool layers a features extraction head.
Revised: We call the connection of convolutional layers and pool layers a features extraction head.

Original: The features extraction head may or may not be connected to a fully connected layer(FCL) based on the application of the deep model.
Revised: The features extraction head may or may not be connected to a fully-connected layer based on the application of the deep model.

Original: One important feature of CNLs that the weights are shared for creating the features.
Revised: One important feature of convolutional layers that the weights are shared for creating the features.

Original: By having this feature of shared weights, CNLs does not strongly contribute to the total parameter size of the deep model unlike the contribution of the FCL.
Revised: By having this feature of shared weights, convolutional layers does not strongly contribute to the total parameter size of the deep model unlike the contribution of the fully-connected layer.

Original: In this work we show empirically, visually and trough a similarity scores analysis that replacing ordinary CNL by an Inner Ensemble average (IEA) of CNLs in a CNN can reduce the variance of this model.
Revised: In this work, we show empirically, visually and through similarity score analysis that replacing ordinary convolutional layer by an Inner Ensemble Average (IEA) of convolutional layers in a CNN can reduce the variance of this model.

Original: To the best of our knowledge we are the ﬁrst to introduce the usage ofensemble average within a CNN architecture instead of combining multiple trained CNNs.
Revised: We propose it as a convenient technique to improve the performance of CNN predictive models.

Original: This work is organized as fellow, section 2 deﬁnes IEA mathematically and discusses the suitable number of ensembles needed.
Revised: This work is organized as follows, section 3 deﬁnes IEA mathematically.

Original: Section 3 shows the experiments performed on different benchmark datasets like MNIST dataset LeCun (1998), rotated-MNIST dataset Larochelle et al. (2007) and CIFAR-10 dataset Krizhevsky & Hinton (2009) using well-known deep CNNs architectures.
Revised: Section 4 shows the experiments performed on different benchmark datasets including MNIST(LeCun, 1998), rotatedMNIST (Larochelle et al., 2007), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009) using well-known deep CNN architectures.

Original: We show the results of CNL only models, IEA of CNLs models and ensemble of both techniques.
Revised: We show the results of convolutional layer only models, IEA of convolutional layers models and ensemble of both techniques.

Original: In[Figure] sections 4 a visual analysis of the  generated from  is performed.
Revised: In sections 5 a visual analysis of the features generated from IEA usage is performed.

Original: Also, in sectionwe introduce a metric to measure the similarity between the features generated from IEA and CNLs.
Revised: Also, in section 5 we introduce a metric to measure the similarity between the features generated from IEA and convolutional layers.  R ELATED W ORK

Revised: Ensemble is one of the meta-algorithms that combines a set of independently trained networks into one predictive model in order to improve performance.

Revised: Other methods include bagging and boosting that reduce predictive variance and bias (Opitz & Maclin, 1999).

Revised: Ensemble by averaging several models is one of the most common approaches and extensively adopted in modern deep learning practice, especially in related contests (Russakovsky et al., 2015; Rajpurkar et al., 2016).

Revised: One variation is to ensemble the same model at different training epochs (Qiu et al., 2014).

Original: I NNER E NSEMBLE A VERAGE (IEA) 2. IEA D EFINITION
Revised: Though inspired by ensemble methods that average independent networks, the proposed IEA structure is fundamentally different in that a) it replicates small units, namely convolutional layers, within CNN structures and b) the replications are trained jointly. Our work is reminiscent of Maxout (Goodfellow et al., 2013), which replicates weight matrices and take the maximum from the features produce by those linear layers. Maxout can be considered as a special activation method that behaves as a learnable piecewise-linear function. IEA differs from Maxout since IEA averages convolutional layers after non-linearity layers (Figure 1).

Revised: While

Revised: Maxout is designed to approximate any activation function, our intuition is as follows: an ensemble of models are used essentially to reduce variance in other terms to increase models generalization.

Revised: We hypothesize that the overall variance of the model can be decomposed into sub-layers within the model and each layer contributes somehow into this variance.

Revised: If we used inner ensemble average we will reduce the sub-variances of each layer resulting in an overall variance reduction.

Revised: Therefore, we primarily explore the averaging method in our setting.

Original: The IEA concept  can be applied to a CNN architecture by replacing any CNL ( C layer ) with average ensemble of CNLs.
Revised: ResNet (He et al., 2015) and its variants (Huang et al., 2017; Szegedy et al., 2017) have become standard for recognition tasks. One can argue that the concept of IEA bears some similarity with the additive behaviors of ResNet and variants (He et al., 2016). Wide Resnet (Zagoruyko & Komodakis, 2016) explores the idea that convolutional layers can have larger dimension while reducing the overall model depth. ResNext (Xie et al., 2017) also makes the model wider by introducing complicated pathways within Resnet blocks and achieved competitive results with more shallow networks.

Original: In any deep CNN architecture to use the IEA concept the C layer is replaced by 
Revised: IEA further exploits the possibility of increasing block width by simply plugging into existing models, replicating any convolutional layers within IEA. In our paper, Wide ResNet and ResNext with IEA are intensively experimented and analyzed in comparison with regular ones. Some other widely-used CNN architectures are also explored.

Original:  C IEA .
Revised: The IEA concept can be applied to a CNN architecture by replacing any convolutional layer ( C layer ) with average ensemble of several replications of a convolutional layer (along with the following batch normalization (Ioffe & Szegedy, 2015) and activation layer, which means averaging is performed after the non-linearity). In any deep CNN architecture to use the IEA concept the C layer is replaced by C IEA .

Original: The C IEA is deﬁned as follows:
Revised: The C IEA is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where m = { x | x ∈ N + −{ 1 }} is the number of inner CNLs which is a design choice.
Revised: where m = { x | x ∈ N + − { 1 }} is the number of inner convolutional layers which is a design choice.

Original: An illustration of IEA concept within a CNN is found in ﬁgure 2.
Revised: An illustration of IEA concept within a CNN is found in Figure 1.

Original: When using IEA, the same settings of the replaced CNL are applied to each IEA element individually.
Revised: When using IEA, the same settings of the replaced convolutional layer are applied to each IEA element individually.

Original: 2. I NFLUENCE OF THE NUMBER OF ENSEMBLES M

Original: We executed a set of tests using one, two and three layers deep CNN models that were trained over the rotated-MNIST dataset to ﬁnd the right number of inner ensembles m .

Original: The conﬁguration of training settings is mentioned in section 3.1.

Original: Each CNL is replaced by an IEA of CNLs.

Original: The m in each IEA layer has been set to integers values ranging from 1 to 30.

Original: A total number of 90 models were trained.

Original: Figure 2 shows the inﬂuence of m on the accuracy of the CNNs.

Original: The accuracy of the CNN increases for low values of m , and saturates or decreases at some point (in the present case

Original: when m (cid:39) 7 ).

Original: [Equation]

Original: [Figure]

Original: Note that this threshold could be a function of several model hyper-parameters (including kernel size, number of channels.... ).

Original:  E XPERIMENTS 
Revised:  E XPERIMENTS AND A NALYSIS

Original: In the following section, we empirically evaluate the performance of IEA CNLS versus ordinary CNL.
Revised: In the following section, we empirically evaluate the performance of IEA convolutional layers versus ordinary convolutional layer.

Original: In all tests, the number of inner ensembles is set to m = 3 , to speed up computations.
Revised: In all tests, the number of inner ensembles is set to m = 3 , to speed up computations.

Original: We also compare the results of average of an ensemble of models trained using CNL and an ensemble of model trained using IEA of CNLs.
Revised: We also compare the results of average of an ensemble of models trained using convolutional layer and an ensemble of model trained using IEA of convolutional layers.

Original: The ensemble of IEA of CNLs can be described as an outer and inner ensemble in a CNN. 
Revised: The ensemble of IEA of convolutional layers can be described as an outer and inner ensemble in a CNN. [Figure] addition [Figure]

Original: 3. P ERFORMANCE ANALYSIS ON THE MNIST AND ROTATED -MNIST DATASETS
Revised: IEA average conv BN ReLU conv BN

Revised: ReLU conv

Revised: BN

Revised: ReLU

Revised: IEA average conv

Revised: BN

Revised: [Figure]

Revised: BN

Revised: [Figure]

Revised: BN addition

Revised: ReLU x l+

Original:  The MNIST dataset is one of the earliest datasets used in computer vision and deep learning for evaluating different models performance.
Revised: [Figure] x l 4. MNIST AND R OTATED -MNIST D ATASETS The MNIST dataset is one of the earliest datasets used in computer vision and deep learning for evaluating different models performance.

Original: The MNIST dataset contains 60,000 training set examples and a test set of 10,000 examples of hand written digits.
Revised: The MNIST dataset contains 60,000 training set examples and a test set of 10,000 examples of hand written digits.

Original: The rotated-MNIST dataset containsrandomly rotated handwritten digits.
Revised: The rotated-MNIST dataset containsrandomly rotated handwritten digits.

Original: The rotated-MNIST dataset is split into a training and test sets of size 50000 and 12000 respectively.
Revised: The rotated-MNIST dataset is split into a training and test sets of size 50000 and 12000 respectively.

Original: Several models were trained on both datasets.
Revised: We assembled several vanilla CNN models and trained on both datasets.

Original: We varied the number of layers in which we control how deep these models are.

Original: Each model contains several layers.
Revised: Each model contains one to three layers.

Original: Each layer is composed of a  convolution layer batch normalized Ioffe & Szegedy (2015) andfollowed by a rectiﬁed linear unit (ReLU)Hahnloser et al. (2000) activation function.
Revised: We deﬁne a convolutional layer as composition of a weight matrix that applies a 2D convolution over an input signal, the following batch normalization layer and rectiﬁed linear unit (ReLU) activation function (Hahnloser et al., 2000).

Original: Then a max pool layer is stacked.
Revised: Then a max-pooling layer is stacked after each convolutional layer.

Original: After stacking the desired number of layers an average pool layer is used before the connection to a classiﬁcation head. 
Revised: Finally, an average-pooling layer is connected before plugging into a classiﬁcation head. We then expand these small CNN models by replicating any convolutional layers by three and adding an average pooling layer, making it a IEA with m = 3 . Accordingly, we use m = 1 to represent regular convolutional layers.

Original:  The models were trained using stochastic gradient descent (SGD) Bottou (2010) optimization algorithm with a momentum Polyak (1964) set to 0.9 and weight decay Krogh & Hertz (1992) set to 5e-4.
Revised: Therefore, we have six different models to test on MNIST/rotated-MNIST datasets. The models were trained using stochastic gradient descent (SGD) (Bottou, 2010) optimization algorithm with a momentum (Polyak, 1964) set to 0.9 and weight decay (Krogh & Hertz, 1992) set to 5e-4.

Original: The initial learning rate is 0.1 and it was divided by 10 for every 100 training epochs.
Revised: The initial learning rate is 0.1 and it was divided by 10 for every 100 training epochs.

Original: The total number of training epochs is 350.
Revised: The total number of training epochs is 350.

Original: The training datasets were not augmented, and the same initialization settings were kept the same between the models.
Revised: The training datasets were not augmented, and the same initialization settings were kept the same between the models.

Original: Table 1 describes the results on the MNIST test dataset.
Revised: Table 1 describes the results on the MNIST test dataset.

Original: The usage of IEA of CNLs leads to an increase in performance over CNL only models.
Revised: The usage of IEA of convolutional layers leads to an increase in performance over convolutional layer only models.

Original: It can be seen in the case of one-layer deep model that the error rate of the model is decreased by 0.18%.
Revised: It can be seen in the case of one-layer deep model that the error rate of the model is decreased by 0.18%.

Original: By going deeper with one more layer, the mean error rate improved by 0.76% with just two IEA of CNLs.
Revised: By going deeper with one more layer, the mean error rate improved by 0.76% with just two IEA of convolutional layers.

Original: We achieve a mean error rate of 0.45% on the MNIST dataset by just using three layers deep model.
Revised: We achieve a mean error rate of 0.45% on the MNIST dataset by just using three layers deep model.

Revised: Even with replications inside the architecture, we can still exploit ensemble of several IEA models to increase the predictive power.

Original: Table 2 shows the  average ensemble of three IEA of CNLs model performance versus an ensemble of three CNL only models. 
Revised: Table 2 shows the performance of average ensemble of three IEA models versus an ensemble of three convolutional layer only models. To distinguish from inner ensemble , we use k to indicate the number of models in an outer ensemble of identical, yet independently trained networks. In our case, k is 3 .

Original: In the case of one-layer deep model, both ensemble of IEA of CNLs and CNL only model show the same performance.
Revised: In the case of one-layered deep model, both ensemble of IEA of convolutional layers and convolutional layer only model show the same performance.

Original: The ensemble of IEA models shows a better performance than the ensemble of CNL only based model when two or three layers are used.
Revised: The ensemble of IEA models shows a better performance than the ensemble of convolutional layer only based model when two or three layers are used.

Original:  Table 3 describes the rotated-MNIST test dataset results.
Revised: [Table] [Table] Table 3 describes the rotated-MNIST test dataset results.

Original: In this case,the usage of IEA of CNLs still leads to signiﬁcant improvements over CNL only models.
Revised: In this case,the usage of IEA of convolutional layers still leads to signiﬁcant improvements over convolutional layer only models.

Original: It can be seen in the case of one-layer model that the IEA of CNLs model signiﬁcantly improve the accuracy compared to the CNL only model.
Revised: It can be seen in the case of one-layer model that the IEA of convolutional layers model signiﬁcantly improve the accuracy compared to the convolutional layer only model.

Original: The performance increases when more layers are added (deeper model).
Revised: The performance increases when more layers are added (deeper model).

Original: We can see a drop of the mean error rate by 0.41% in the case of two layers deep model, and a drop of 0.68% with three layers deep one.
Revised: We can see a drop of the mean error rate by 0.41% in the case of two layers deep model, and a drop of 0.68% with three layers deep one.

Original: These results show the capability of IEA in decreasing the total model variance.
Revised: These results show the capability of IEA in decreasing the total model variance.

Original: Table 4 shows the case of average ensemble of three IEA of CNL models and CNL only based models.
Revised: Table 4 shows the case of average ensemble of three IEA of convolutional layer models and convolutional layer only based models.

Original: In the case of a one layer deep model, the ensemble of CNL models is better than the ensemble of IEA of CNL models.
Revised: In the case of a one layer deep model, the ensemble of convolutional layer models is better than the ensemble of IEA of convolutional layer models.

Original: When more layers are considered, the ensemble of IEA of CNL models outperforms the ensemble of CNL only models, except in the case of two layers model were the ensemble of CNL only models has a slightly better performance.
Revised: When more layers are considered, the ensemble of IEA of convolutional layer models outperforms the ensemble of convolutional layer only models, except in the case of two layers model were the ensemble of convolutional layer only models has a slightly better performance.

Original: We due this behavior to the selection of hyper-parameters.
Revised: We attribute this behavior to the selection of hyper-parameters.

Original: Also, an error rate of 5.33% is achieved using three layers deep model with IEA of CNLs.
Revised: Also, an error rate of 5.33% is achieved using three layers deep model with IEA of convolutional layers.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table] [Table] [Table] P ERFORMANCE ANALYSIS ON THE CIFAR-10 DATASET
Revised: 4. CIFAR-10 D ATASET

Original: The CIFAR-10 dataset consists of 60000 color images of size 32x32 and contains 10 classes.
Revised: The CIFAR-10 dataset consists of 60000 color images of size 32x32 and contains 10 classes.

Original: There are 50000 training images and 10000 test images.
Revised: There are 50000 training images and 10000 test images.

Original: A well-known object detection models like
Revised: Well-known object detection models including

Original: VGG16 Simonyan & Zisserman (2014), residual network with 18 layers (ResNet18) and 101 layers(RestNet101)
Revised: VGG16 (Simonyan & Zisserman, 2014), residual network with 18 layers (ResNet18) and 101 layers

Original: He et al.
Revised: [Table]

Original: Mobilenet Howard et al.
Revised: [Table] [Table]

Original: (2017) and DenselyConnected Convolutional
Revised: 2017), Densely Convolutional

Original: Networks with 121 layers (DenseNet) 
Revised: Networks with 121 layers (DenseNet) (Huang et al

Original: Huang et al. (2016) were trained on the CIFAR-10 dataset.
Revised: [Table] Wide ResNet (Zagoruyko & Komodakis, 2016) and ResNext (Xie et al., 2017) were trained on the CIFAR-10 dataset.

Revised: Note that Wide ResNet and ResNext are trained on the CIFAR-10 dataset that is augmented using image translation and mirroring.

Original: In these models the CNLs were replaced by an IEA CNLs layers.
Revised: In these models the convolutional layers were replaced by an IEA convolutional layers layers.

Original: The training was done using a non-augmented training samples.
Revised: The training was done using a non-augmented training samples.

Original: The training conﬁguration is the same as the conﬁgurations mentioned in section 3.1.
Revised: The training conﬁguration is the same as the conﬁgurations mentioned in section 4.1. The training and validation curves can be found in the supplementary materials.

Original: Table 5 shows an overall improvement in classiﬁcation mean error rates on CIFAR-10 dataset.
Revised: Table 5 shows an overall improvement in classiﬁcation mean error rates on CIFAR-10 dataset.

Original: VGG had a 0.64% mean error rate improvement by using IEA of CNLs.
Revised: VGG had a 0.64% mean error rate improvement by using IEA of convolutional layers.

Original: ResNet18 and ResNet101 were improved by 1.0%, 0.52% mean error rates respectively by using IEA compared to CNL.
Revised: ResNet18 and ResNet101 were improved by 1.0%, 0.52% mean error rates respectively by using IEA compared to convolutional layer.

Original: A signiﬁcant improvement is seen in MobileNet by 3.09% error rate when using IEA, though IEA leads to an increase in total model run-time, which goes against the main purpose of MobileNet.
Revised: A signiﬁcant improvement is seen in MobileNet by 3.09% error rate when using IEA.

Original: DenseNet shows an improvement of 0.13% by using IEA of CNLs.
Revised: DenseNet shows an improvement of 0.13% by using IEA of convolutional layers. Wide

Original:  To test the effect of models ensemble, table 6 shows an ensemble of previously mentioned models.
Revised: ResNet and ResNext also improves. To test the effect of models ensemble, Table 6 shows an ensemble of previously mentioned models.

Original: We average an ensemble of three models using IEA of CNLs and another three models using CNLs only.
Revised: We average an ensemble of three models using IEA of convolutional layers and another three models using convolutional layers only.

Original: The ensemble of IEA of CNLs models show a better performance than the CNL only models.
Revised: The ensemble of IEA of convolutional layers models show a better performance than the convolutional layer only models.

Original: [Table] 
Revised: [Table] [Table]

Revised: 4. CIFAR-100 D ATASET

Revised: The property images in the CIFAR-100 dataset is identical to that of the CIFAR-

Original: [Table]
Revised: [Table]

Revised: except that it has 100 classes containing 600 images each.

Revised: There are 500 training images and 100 testing images per class.

Original: V ISUALIZATION AND ANALYSIS OF IEA FEATURES
Revised: Compared to previously investigated datasets, the CIFAR-100 dataset is much larger and models can truly manifest their power when benchmarked against it. Note that the CIFAR-100 dataset here is augmented using image translation and/or mirroring. In this section, we plug IEA into Wide ResNet and ResNext and train them on the CIFAR-100. In

Revised: Table 7, our experiments show that even if our implementation of both networks perform worse than the original paper, our Wide ResNet and ResNext with IEA achieve considerable results.

Revised: Spceciﬁcally,

Revised: Wide ResNet with IEA decreases the error rate from 18.85% (Zagoruyko & Komodakis, 2016) to 18.03% on CIFAR-100.

Revised: ResNext with IEA achieves performance on par with ResNext (Xie et al., 2017).

Revised: Further, we use outer ensemble to combine individual networks and show in Table 8 that

Revised: IEA will by no means exhaust the power of variance reduction and outer ensemble can still obtain considerable performance boost.

Revised: Our results conﬁrm that IEA can still boost the model performance on large scale datasets.

Revised: The training and validation curves can be found in the supplementary materials.

Revised: Also, in order to understand the computational headache of using IEA table 9 states the inference time per image in milliseconds over CIFARA-100 validation data set.

Revised: It can be seen that m =

Revised: Wide

Original: To understand how IEA works and produces better results than ordinary CNL, we visualized the features generated by both.
Revised: ResNet inference time is the same as m = 1 ResNext. Meanwhile, we can tell that IEA introduces almost m times computational headache when used and this is a future research point. [Table] [Table]

Original: Figure 3 shows different features generated by IEA CNLs and CNL.
Revised: [Table]  V ISUALIZATION AND ANALYSIS OF IEA FEATURES 5. V ISUAL INTERPRETATION OF IEA FEATURES To understand how IEA works and produces better results than ordinary convolutional layer, we visualized the features generated by both. Table 10 shows different features generated by IEA layers and convolutional layer.

Original: It is noticeable that the features generated by IEA tend to be more unique and different from each other, unlike CNLs where some features appear to be identical.
Revised: It is noticeable that the features generated by IEA tend to be more unique and different from each other, unlike convolutional layers where some features appear to be identical.

Original: Also, in section (a) of ﬁgure 3 where the deep model is only 1 layer deep, there exist some IEA features that are associated with zero weight after training.
Revised: Also, in the ﬁrst row of Table 10 where the deep model is only 1 layer deep, there exist some IEA features that are associated with zero weight after training.

Original: This demonstrates that IEA removed some unnecessary features from the model, which helps improving robustness. 
Revised: This demonstrates that IEA removed some unnecessary features from the model, which helps improving robustness. [Table] [Table]

Original: We use similarity scores λ to validate the visual interpretation of the IEA features.
Revised: We use similarity scores λ to validate the visual interpretation of the IEA features.

Original: The similarity score measures how similar an image to another image.
Revised: The similarity score measures how similar an image to another image.

Original: If both image of comparison are the same, the similarity score value will be zero.
Revised: If both image of comparison are the same, the similarity score value will be zero.

Original: The more difference between the images the more the value of similarity score increases till it reaches the maximum value which is 1.
Revised: The more difference between the images the more the value of similarity score increases till it reaches the maximum value which is 1.

Original: Thus, the usage of similarity score is handful to measure how unique a feature compared to other features produced by the same layer.
Revised: Thus, the usage of similarity score is handful to measure how unique a feature compared to other features produced by the same layer.

Original: We introduce the mean sum of similarity score (mss-score) µ λ as follows: for each n features f in a layer the mss-score is deﬁned as:
Revised: We introduce the mean sum of similarity score (mss-score) µ λ as follows: for each n features f in a layer the mss-score is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The higher the mss-score is, the more unique the features produced by the model are.
Revised: The higher the mss-score is, the more unique the features produced by the model are.

Original: For the similarity score measurement, we used the similarity score model introduced by Zhang et al.
Revised: For the similarity score measurement, we used the similarity score model introduced by (Zhang et al., 2018)

Original: (2018) which is proven to outperform any previous similarity score measurements methods.
Revised: [Table] which is proven to outperform any previous similarity score measurements methods.

Original: In table 7, the IEA and CNL mss-scores were evaluated on a batch of 100 validation samples from the rotated-MNIST dataset.
Revised: In Table 11, the IEA and convolutional layer mss-scores were evaluated on a batch of 100 validation samples from the rotated-MNIST dataset.

Original: The IEAs mss-scores are always greater than the CNL mss-scores.
Revised: The IEAs mss-scores are always greater than the convolutional layer mss-scores.

Original: This indicates that the usage of IEA produces more unique features compared to CNL features, and it conﬁrms our visual analysis. 
Revised: This indicates that the usage of IEA produces more unique features compared to convolutional layer features, and it conﬁrms our visual analysis. 5. C OMPARISON WITH M AX O UT FEATURES

Original: [Table]
Revised: To develop the understanding of the features generated by using IEA, we compare IEA features against the features generated by MaxOut. A one layer vanilla CNN with 48 features channels were trained using MaxOut, the max parameter of MaxOut is set to 3 leading to a network that is similar in the parameters size to the IEA one. By setting the max parameter to 3, the 48 features turns into 16 features, this fortiﬁes a fair comparison with the IEA features on the same model. The training settings were the same as mentioned in section 4.1 but with a lower learning rate set to 0.001 initially.

Revised: The trained MaxOut model had an error rate of 21 .

Revised: 86% on the rotated-MNIST validation dataset.

Revised: Visuallay, we can tell that the MaxOut features are more unique than the Vanilla CNN features but they have some similarity in between.

Revised: Also, MaxOut did not produce such a unique set of features like

Revised: IEA.

Revised: One expects that the MaxOut features mss-score to be larger than Vanilla CNN ones, when we computed it had a value of 0 .

Revised: 01084 almost the third of the mss-score value of Vanilla CNN mss-score obtained from table 11.

Revised: This was surprisingly to us.

Revised: To investigate this behavior, We state these two facts.

Revised: Firstly, we calculate the mss-score over a validation batch of 100 images.

Revised: Secondly, because the MaxOut chooses the max ﬁlter between groups this will results in a different model for each input.

Revised: This suggest that each input image will have a different similarity score as the model behavior changes based on the input.

Revised: This could lead in total to obtain a low mss-score, also it suggests that it is a drawback of MaxOut compared to IEA.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: IEA is a concept that is simple but powerful.
Revised: IEA is a concept that is simple but powerful.

Original: It helps the CNN architecture to increase its prediction power by forcing the model to produce more unique features.
Revised: It helps the CNN architecture to increase its prediction power by forcing the model to produce more unique features.

Original: The cost of using IEA in a CNN is on the model parameter size, yet it’s not a signiﬁcance cost to the improvement in the performance.
Revised: The cost of using IEA in a CNN is on the model parameter size, yet it’s not a signiﬁcance cost to the improvement in the performance.

Original: We showed empirically, visually and by using a similarity scores that the usage of IEA improves the CNNs accuracy and produces unique features.
Revised: We showed empirically, visually and by using a similarity scores that the usage of IEA improves the CNN accuracy and produces unique features.

Original: Also, the ensemble of IEA of CNLs models outperforms the ensemble of CNL only model which is a method of inner and outer ensemble.
Revised: Also, the ensemble of IEA of convolutional layers models outperforms the ensemble of convolutional layer only model which is a method of inner and outer ensemble.

Original: We recommend the usage of IEA where it applies, and a further study of other methods of ensembles shall be conducted.
Revised: We recommend the usage of IEA where it applies, and a further study of other methods of ensembles shall be conducted.

Original: Input
Revised: Our future work also include minimizing the inference time of IEA by ﬁnding a criteria for disabling the non-important features and expanding our tests to other not-vision based domains. Supplementary Materials T RAINING AND VALIDATION CURVES [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: [Figure]