Original: E XCITATION D ROPOUT : E NCOURAGING P LASTICITY IN D EEP N EURAL N ETWORKS
Revised: E XCITATION D ROPOUT : E NCOURAGING P LASTICITY IN D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a guided dropout regularizer for deep networks based on the evidence of a network prediction: the ﬁring of neurons in speciﬁc paths.
Revised: We propose a guided dropout regularizer for deep networks based on the evidence of a network prediction: the ﬁring of neurons in speciﬁc paths.

Original: In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout.
Revised: In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout.

Original: In essence, we dropout with higher probability those neurons which contribute more to decision making at training time.
Revised: In essence, we dropout with higher probability those neurons which contribute more to decision making at training time.

Original: This approach penalizes high saliency neurons that are most relevant for model prediction, i.e. those having stronger evidence.
Revised: This approach penalizes high saliency neurons that are most relevant for model prediction, i.e. those having stronger evidence.

Original: By dropping such high-saliency neurons, the network is forced to learn alternative paths in order to maintain loss minimization, resulting in a plasticity-like behavior, a characteristic of human brains too.
Revised: By dropping such high-saliency neurons, the network is forced to learn alternative paths in order to maintain loss minimization, resulting in a plasticity-like behavior, a characteristic of human brains too.

Original: We demonstrate better generalization ability, an increased utilization of network neurons, and a higher resilience to network compression using several metrics over four image/video recognition benchmarks.
Revised: We demonstrate better generalization ability, an increased utilization of network neurons, and a higher resilience to network compression using several metrics over four image/video recognition benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dropout [Hinton et al.
Revised: Dropout [Hinton et al.

Original: ; Srivastava et al.
Revised: ; Srivastava et al.

Original: (2014)] is a classical regularization technique that is used in many state-of-the-art deep neural networks, typically applied to fully-connected layers.
Revised: (2014)] is a classical regularization technique that is used in many state-of-the-art deep neural networks, typically applied to fully-connected layers.

Original: Standard Dropout selects a fraction of neurons to randomly drop out by zeroing their forward signal.
Revised: Standard Dropout selects a fraction of neurons to randomly drop out by zeroing their forward signal.

Original: In this work, we propose a scheme for biasing this selection.
Revised: In this work, we propose a scheme for biasing this selection.

Original: Our scheme utilizes the contribution of neurons to the prediction made by the network at a certain training iteration stage.
Revised: Our scheme utilizes the contribution of neurons to the prediction made by the network at a certain training iteration stage.

Original: Dropout can be interpreted as model averaging technique that avoids overﬁtting on training data, allowing for better generalization on unseen test data.
Revised: Dropout can be interpreted as model averaging technique that avoids overﬁtting on training data, allowing for better generalization on unseen test data.

Original: A recent variant of dropout that targets improved generalization ability is Curriculum Dropout [Morerio et al.
Revised: A recent variant of dropout that targets improved generalization ability is Curriculum Dropout [Morerio et al.

Original: It targets adjusting the dropout rate by exponentially increasing the unit suppression rate during training, answering the question How many neurons to drop out over time?
Revised: It targets adjusting the dropout rate by exponentially increasing the unit suppression rate during training, answering the question How many neurons to drop out over time?

Original: Like Standard Dropout [Hinton et al.
Revised: Like Standard Dropout [Hinton et al.

Original: Srivastava et al.
Revised: Srivastava et al.

Original: , Curriculum Dropout selects the neurons to be dropped randomly.
Revised: , Curriculum Dropout selects the neurons to be dropped randomly.

Original: In this work, however, we target at determining how the dropped neurons are selected, answering the question Which neurons to drop out?
Revised: In this work, however, we target at determining how the dropped neurons are selected, answering the question Which neurons to drop out?

Original: Our approach is inspired by brain plasticity [Hebb (2005); Song et al.
Revised: Our approach is inspired by brain plasticity [Hebb (2005); Song et al.

Original: Mittal et al.
Revised: Mittal et al.

Original: Miconi et al.
Revised: Miconi et al.

Original: We deliberately, and temporarily, paralyze/injure neurons to enforce learning alternative paths in a deep network.
Revised: We deliberately, and temporarily, paralyze/injure neurons to enforce learning alternative paths in a deep network.

Original: At training time, neurons that are more relevant to the current prediction are given a higher dropout probability.
Revised: At training time, neurons that are more relevant to the current prediction are given a higher dropout probability.

Original: The relevance of a neuron for making a certain prediction is quantiﬁed using Excitation Backprop, a top-down saliency approach proposed by Zhang et al. (2016).
Revised: The relevance of a neuron for making a certain prediction is quantiﬁed using Excitation Backprop, a top-down saliency approach proposed by Zhang et al. (2016).

Original: Excitation Backprop conveniently yields a probability distribution at each layer that reﬂects neuron saliency, or neuron contribution to the prediction being made.
Revised: Excitation Backprop conveniently yields a probability distribution at each layer that reﬂects neuron saliency, or neuron contribution to the prediction being made.

Original: This is utilized in the pipeline of our approach, named Excitation Dropout, which is summarized in Fig.
Revised: This is utilized in the pipeline of our approach, named Excitation Dropout, which is summarized in Fig.

Original: In particular, we study how this approach improves generalization through utilizing more network’s neurons for image classiﬁcation.
Revised: In particular, we study how this approach improves generalization through utilizing more network’s neurons for image classiﬁcation.

Original: We report an increased recognition rate for both CNN models that are ﬁne-tuned and trained from scratch.
Revised: We report an increased recognition rate for both CNN models that are ﬁne-tuned and trained from scratch.

Original: This improvement is validated on four image/video recognition datasets, and ranges from 1.1% - 6.3% over state-of-the-art Curriculum Dropout.
Revised: This improvement is validated on four image/video recognition datasets, and ranges from 1.1% - 6.3% over state-of-the-art Curriculum Dropout.

Original: Next, we examine the effect of our approach on network utilization.
Revised: Next, we examine the effect of our approach on network utilization.

Original: Mittal et al.
Revised: Mittal et al.

Original: (2018) and Ma et al.
Revised: (2018) and Ma et al.

Original: (2017) introduce metrics that measure network utilization.
Revised: (2017) introduce metrics that measure network utilization.

Original: We show a consistent increased network utilization using Excitation Dropout on four image/video recognition datasets.
Revised: We show a consistent increased network utilization using Excitation Dropout on four image/video recognition datasets.

Original: For example, averaged over all four benchmarks, we get 76.55% reduction in conservative ﬁlters, ﬁlters whose parameters do not change signiﬁcantly during training, as compared to Standard Dropout.
Revised: For example, averaged over all four benchmarks, we get 76.55% reduction in conservative ﬁlters, ﬁlters whose parameters do not change signiﬁcantly during training, as compared to Standard Dropout.

Original: Finally, we study network resilience to neuron dropping at test time.
Revised: Finally, we study network resilience to neuron dropping at test time.

Original: We observe that training with
Revised: We observe that training with

Original: Excitation Dropout leads to models that are a lot more robust when layers are shrunk/compressed by removing units.
Revised: Excitation Dropout leads to models that are a lot more robust when layers are shrunk/compressed by removing units.

Original: We demonstrate this when dropping the most relevant neurons, the least relevant neurons, and with a random dropping selection.
Revised: We demonstrate this when dropping the most relevant neurons, the least relevant neurons, and with a random dropping selection.

Original: This can be quite desirable for compressing/distilling [Hinton et al.
Revised: This can be quite desirable for compressing/distilling [Hinton et al.

Original: (2015)] a model, e.g. for deployment on mobile devices.
Revised: (2015)] a model, e.g. for deployment on mobile devices.

Original: In summary, by encouraging plasticity-like behavior, our contributions are threefold:
Revised: In summary, by encouraging plasticity-like behavior, our contributions are threefold:

Original: Better generalization on test data.
Revised: Better generalization on test data.

Original: Higher utilization of network neurons.
Revised: Higher utilization of network neurons.

Original: Resilience to network compression.
Revised: Resilience to network compression.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Dropout was ﬁrst introduced by Hinton et al.
Revised: Dropout was ﬁrst introduced by Hinton et al.

Original: (2012) and Srivastava et al.
Revised: (2012) and Srivastava et al.

Original: as a way to prevent neural units from co-adapting too much on the training data by randomly omitting subsets of neurons at each iteration of the training phase.
Revised: as a way to prevent neural units from co-adapting too much on the training data by randomly omitting subsets of neurons at each iteration of the training phase.

Original: Some follow-up works have explored different schemes for determining how much dropout is applied to neurons/weights.
Revised: Some follow-up works have explored different schemes for determining how much dropout is applied to neurons/weights.

Original: Wager et al.
Revised: Wager et al.

Original: (2013) described the dropout mechanism in terms of an adaptive regularization, establishing a connection to the AdaGrad algorithm.
Revised: (2013) described the dropout mechanism in terms of an adaptive regularization, establishing a connection to the AdaGrad algorithm.

Original: Inspired by information theoretic principles, Achille & Soatto (2018) propose Information Dropout, a generalization dropout which can be automatically adapted to the data.
Revised: Inspired by information theoretic principles, Achille & Soatto (2018) propose Information Dropout, a generalization dropout which can be automatically adapted to the data.

Original: Kingma et al.
Revised: Kingma et al.

Original: (2015) showed that a relationship between dropout and Bayesian inference can be extended when the dropout rates are directly learned from the data.
Revised: (2015) showed that a relationship between dropout and Bayesian inference can be extended when the dropout rates are directly learned from the data.

Original: Kang et al.
Revised: Kang et al.

Original: (2017) introduces Shakeout which instead of randomly discarding units as dropout does, it randomly enhances or reverses each units contribution to the next layer.
Revised: (2017) introduces Shakeout which instead of randomly discarding units as dropout does, it randomly enhances or reverses each units contribution to the next layer.

Original: Wan et al.
Revised: Wan et al.

Original: (2013) introduced the DropConnect framework, adding dynamic sparsity on the weights of a deep model.
Revised: (2013) introduced the DropConnect framework, adding dynamic sparsity on the weights of a deep model.

Original: DropConnect generalized Standard Dropout by randomly dropping the weights rather than the neuron activations in the network.
Revised: DropConnect generalized Standard Dropout by randomly dropping the weights rather than the neuron activations in the network.

Original: Rennie et al.
Revised: Rennie et al.

Original: (2014) proposed a time scheduling for the retaining probability for the neurons in the network.
Revised: (2014) proposed a time scheduling for the retaining probability for the neurons in the network.

Original: The presented adaptive regularization scheme smoothly decreased in time the number of neurons turned off during training.
Revised: The presented adaptive regularization scheme smoothly decreased in time the number of neurons turned off during training.

Original: Recently, Morerio et al.
Revised: Recently, Morerio et al.

Original: (2017) proposed Curriculum Dropout to adjust the dropout rate in the opposite direction, exponentially increasing unit suppression rate during training, leading to a better generalization on unseen data.
Revised: (2017) proposed Curriculum Dropout to adjust the dropout rate in the opposite direction, exponentially increasing unit suppression rate during training, leading to a better generalization on unseen data.

Original: Other works focus on which neurons to drop out.
Revised: Other works focus on which neurons to drop out.

Original: Dropout is usually applied to fully-connected layers of a deep network.
Revised: Dropout is usually applied to fully-connected layers of a deep network.

Original: Conversely, Wu & Gu (2015) studied the effect of dropout in convolutional and pooling layers.
Revised: Conversely, Wu & Gu (2015) studied the effect of dropout in convolutional and pooling layers.

Original: The selection of neurons to drop depends on the layer where they reside.
Revised: The selection of neurons to drop depends on the layer where they reside.

Original: In contrast, we select neurons within a layer based on their contribution.
Revised: In contrast, we select neurons within a layer based on their contribution.

Original: Wang & Manning (2013) demonstrate that sampling neurons from a Gaussian approximation gave an order of magnitude speedup and more stability during training.
Revised: Wang & Manning (2013) demonstrate that sampling neurons from a Gaussian approximation gave an order of magnitude speedup and more stability during training.

Original: Li et al.
Revised: Li et al.

Original: (2016) proposed to use multinomial sampling for dropout, i.e. keeping neurons according to a multinomial distribution with speciﬁc probabilities for different neurons.
Revised: (2016) proposed to use multinomial sampling for dropout, i.e. keeping neurons according to a multinomial distribution with speciﬁc probabilities for different neurons.

Original: Ba & Frey (2013) jointly trained a binary belief network with a neural network to regularize its hidden units by selectively setting activations to zero accordingly to their magnitude.
Revised: Ba & Frey (2013) jointly trained a binary belief network with a neural network to regularize its hidden units by selectively setting activations to zero accordingly to their magnitude.

Original: While this takes into consideration the magnitude of the forward activations, it does not take into consideration the relationship of these activations to the ground-truth.
Revised: While this takes into consideration the magnitude of the forward activations, it does not take into consideration the relationship of these activations to the ground-truth.

Original: In contrast, we drop neurons based on how they contribute to a network’s decision.
Revised: In contrast, we drop neurons based on how they contribute to a network’s decision.

Original: We compare our results against Morerio et al.
Revised: We compare our results against Morerio et al.

Original: To the best of our knowledge, we are the ﬁrst to probabilistically select neurons to dropout based on their task-relevance.
Revised: To the best of our knowledge, we are the ﬁrst to probabilistically select neurons to dropout based on their task-relevance.

Original:  M ETHOD 3. B ACKGROUND
Revised:  M ETHOD 3. B ACKGROUND

Original: Saliency maps that quantize the importance of class-speciﬁc neurons for an input image are instrumental to our proposed scheme.
Revised: Saliency maps that quantize the importance of class-speciﬁc neurons for an input image are instrumental to our proposed scheme.

Original: Popular approaches include Class Activation Maps (CAM)
Revised: Popular approaches include Class Activation Maps (CAM)

Original: [Zhou et al.
Revised: [Zhou et al.

Original: Gradient-weighted Class Activation Mapping (Grad-CAM)
Revised: Gradient-weighted Class Activation Mapping (Grad-CAM)

Original: [Selvaraju et al.
Revised: [Selvaraju et al.

Original: (2017)], and Excitation Backprop (EB)
Revised: (2017)], and Excitation Backprop (EB)

Original: [Zhang et al.
Revised: [Zhang et al.

Original: We choose to use EB since it produces a valid probability distribution for each network layer.
Revised: We choose to use EB since it produces a valid probability distribution for each network layer.

Original: In a standard CNN, the forward activation of neuron a j is computed by (cid:98)
Revised: In a standard CNN, the forward activation of neuron a j is computed by (cid:98)

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ( (cid:80) i w ij (cid:98) a
Revised: ( (cid:80) i w ij (cid:98) a

Original: i + b i ) , where (cid:98) a i is the activation coming from the previous layer, φ is a nonlinear activation function, w ij and b i are the weight from neuron i to neuron j and the added bias at layer i , respectively.
Revised: i + b i ) , where (cid:98) a i is the activation coming from the previous layer, φ is a nonlinear activation function, w ij and b i are the weight from neuron i to neuron j and the added bias at layer i , respectively.

Original: EB devises a backpropagation formulation able to reconstruct the evidence used by a deep model to make decisions.
Revised: EB devises a backpropagation formulation able to reconstruct the evidence used by a deep model to make decisions.

Original: It computes the probability of each neuron recursively using conditional probabilities P ( a i | a j ) in a top-down order starting from a probability distribution over the output units, as follows:
Revised: It computes the probability of each neuron recursively using conditional probabilities P ( a i | a j ) in a top-down order starting from a probability distribution over the output units, as follows:

Original: [Equation]
Revised: [Equation]

Original: where P i is the parent node set of a i .
Revised: where P i is the parent node set of a i .

Original: EB passes top-down signals through excitatory connections having non-negative activations, excluding from the competition inhibitory ones.
Revised: EB passes top-down signals through excitatory connections having non-negative activations, excluding from the competition inhibitory ones.

Original: Assuming C j the child node set of a j , for each a i ∈ C j , the conditional winning probability P ( a i | a j ) is deﬁned as
Revised: Assuming C j the child node set of a j , for each a i ∈ C j , the conditional winning probability P ( a i | a j ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Z j is a normalization factor such that (cid:80) a i ∈C j P ( a i | a j ) = 1 .
Revised: where Z j is a normalization factor such that (cid:80) a i ∈C j P ( a i | a j ) = 1 .

Original: Recursively propagating the top-down signal and preserving the sum of backpropagated probabilities, it is possible to highlight the salient neurons in each layer using Eqn. 1, i.e. neurons that mostly contribute to a speciﬁc task.
Revised: Recursively propagating the top-down signal and preserving the sum of backpropagated probabilities, it is possible to highlight the salient neurons in each layer using Eqn. 1, i.e. neurons that mostly contribute to a speciﬁc task.

Original: We will refer to the distribution of P ( a i ) as p EB ( a i ) .
Revised: We will refer to the distribution of P ( a i ) as p EB ( a i ) .

Original: 3. E XCITATION D ROPOUT
Revised: 3. E XCITATION D ROPOUT

Original: In the standard formulation of dropout [Hinton et al.
Revised: In the standard formulation of dropout [Hinton et al.

Original: ; Srivastava et al.
Revised: ; Srivastava et al.

Original: (2014)], the suppression of a neuron in a given layer is modeled by a Bernoulli random variable 0 < p ≤ 1 where p is deﬁned as the probability of retaining a neuron.
Revised: (2014)], the suppression of a neuron in a given layer is modeled by a Bernoulli random variable 0 < p ≤ 1 where p is deﬁned as the probability of retaining a neuron.

Original: Given a speciﬁc layer where dropout is applied, during the training phase, each neuron is turned off with a probability 1 − p .
Revised: Given a speciﬁc layer where dropout is applied, during the training phase, each neuron is turned off with a probability 1 − p .

Original: We argue for a different approach that is guided in the way it selects neurons to be dropped.
Revised: We argue for a different approach that is guided in the way it selects neurons to be dropped.

Original: In a training iteration, certain paths have high excitation contributing to the resulting classiﬁcation, while other regions of the network have low response.
Revised: In a training iteration, certain paths have high excitation contributing to the resulting classiﬁcation, while other regions of the network have low response.

Original: We encourage learning alternative paths (plasticity) through the temporary damaging of the currently highly excited path.
Revised: We encourage learning alternative paths (plasticity) through the temporary damaging of the currently highly excited path.

Original: We re-deﬁne the probability of retaining a neuron as a function of its contribution in the currently highly excited path
Revised: We re-deﬁne the probability of retaining a neuron as a function of its contribution in the currently highly excited path

Original: [Equation]
Revised: [Equation]

Original: whereN is the number of neurons in layer l , p EB is the probability backpropagated through the EB formulation (Eqn. 1) in layer l ,and P is the base probability of retaining a neuron when all neurons are equally contributing to the prediction.
Revised: where p EB is the probability backpropagated through the EB formulation (Eqn. 1) in layer l , P is the base probability of retaining a neuron when all neurons are equally contributing to the prediction and N is the number of neurons in a fully-connected layer l or the number of ﬁlters in a convolutional layer l .

Original: The retaining probability deﬁned in Eqn.
Revised: The retaining probability deﬁned in Eqn.

Original: 3 drops neurons which contribute the most to the recognition of a speciﬁc class, with higher probability.
Revised: 3 drops neurons which contribute the most to the recognition of a speciﬁc class, with higher probability.

Original: Dropping out highly relevant neurons, we retain less relevant ones and thus encourage them to awaken.
Revised: Dropping out highly relevant neurons, we retain less relevant ones and thus encourage them to awaken.

Original: We also study how this compares to dropping the least relevant neurons (Adaptive Dropout by Ba & Frey (2013)) in the Appendix.
Revised: We also study how this compares to dropping the least relevant neurons (Adaptive Dropout by Ba & Frey (2013)) in the Appendix.

Original: Fig.
Revised: Fig.

Original: 2 shows p as a function of p EB .
Revised: 2 shows p as a function of p EB .

Original: To gain some intuition for Eqn. 3, we can look more closely at the graph: 1) If neuron a i has p EB ( a i ) =
Revised: To gain some intuition for Eqn. 3, we can look more closely at the graph: 1) If neuron a i has p EB ( a i ) =

Original: 1 : This results in a retaining probability of p = 0 .
Revised: 1 : This results in a retaining probability of p = 0 .

Original: We do not want to keep a neuron which has a high contribution to the correct label.
Revised: We do not want to keep a neuron which has a high contribution to the correct label.

Original: If neuron a i has p EB ( a i ) = 0 : This results in a retaining probability of p = 1 .
Revised: If neuron a i has p EB ( a i ) = 0 : This results in a retaining probability of p = 1 .

Original: We want to keep a neuron which has not contributed to the correct classiﬁcation of an image.
Revised: We want to keep a neuron which has not contributed to the correct classiﬁcation of an image.

Original: If neuron a i has p EB ( a i ) = 1 /N , i.e. p EB is a uniform probability distribution: This results in a retaining probability p
Revised: If neuron a i has p EB ( a i ) = 1 /N , i.e. p EB is a uniform probability distribution: This results in a retaining probability p

Original: =
Revised: =

Original: P .
Revised: P .

Original: We want to keep a neuron with base probability P since all neurons contribute equally.
Revised: We want to keep a neuron with base probability P since all neurons contribute equally.

Original: Eqn. 3 provides a dropout probability for each neuron, which is then used as the parameter of a
Revised: Eqn. 3 provides a dropout probability for each neuron, which is then used as the parameter of a

Original: Bernoulli distribution giving a binary dropout mask.
Revised: Bernoulli distribution giving a binary dropout mask.

Original: During training, each image in a batch leads to different excitatory connections in the network and therefore has a different p EB distribution, consequently leading to a different dropout mask.
Revised: During training, each image in a batch leads to different excitatory connections in the network and therefore has a different p EB distribution, consequently leading to a different dropout mask.

Original: Fig.
Revised: Fig.

Original: 1 presents the pipeline of Excitation Dropout at training time, and a run-time analysis is presented in the Appendix.
Revised: 1 presents the pipeline of Excitation Dropout at training time, and a run-time analysis is presented in the Appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present how Excitation Dropout(ED) improves the generalization ability on four image/video recognition datasets on different architectures.
Revised: In this section, we present how Excitation Dropout improves the generalization ability on four image/video recognition datasets in fully connected layers of different architectures.

Original: We then present an analysis of how ED affects the utilization of network neurons on the same datasets.
Revised: We then present an analysis of how Excitation Dropout affects the utilization of network neurons on the same datasets.

Original: Finally, we examine the resilience of a model trained using Excitation Dropout to network compression.
Revised: Finally, we examine the resilience of a model trained using Excitation Dropout to network compression.

Original: 4. D ATASETS AND A RCHITECTURES
Revised: 4. D ATASETS AND A RCHITECTURES

Original: We present results on four image/video recognition datasets.
Revised: We present results on four image/video recognition datasets.

Original: Cifar10 and Cifar

Revised: Cifar10 and Cifar

Original: [Krizhevsky (2009)] are image recognition datasets, each consisting of 60000 32 ×
Revised: [Krizhevsky (2009)] are image recognition datasets, each consisting of 60000 32 ×

Original: 32 tiny RGB natural images.
Revised: 32 tiny RGB natural images.

Original: Cifar10 images are distributed over 10 classes with 6000 images per class, and Cifar100 images are distributed over 100 classes with 600 images per class.
Revised: Cifar10 images are distributed over 10 classes with 6000 images per class, and Cifar100 images are distributed over 100 classes with 600 images per class.

Original: Training and test splits contain 50 K and 10 K images, respectively.
Revised: Training and test splits contain 50 K and 10 K images, respectively.

Original: We feed the network with the original image dimensions.
Revised: We feed the network with the original image dimensions.

Original: Caltech [Grifﬁn et al.
Revised: Caltech [Grifﬁn et al.

Original: (2007)] is an image recognition dataset consisting 31000 RGB images divided inclasses.
Revised: (2007)] is an image recognition dataset consisting 31000 RGB images divided in classes.

Original: We consider  50 train images and 20 testing images for each class.
Revised: We consider ﬁve different random splits of 50 train images and 20 testing images for each class.

Original: Images were reshaped to 128 × 128 pixel to feed the network.
Revised: Images were reshaped to 128 × 128 pixel to feed the network.

Original: UCF101 [Soomro et al.
Revised: UCF101 [Soomro et al.

Original: (2012)] is a video action recognition dataset based on 13320 actions belonging to 101 action classes.
Revised: (2012)] is a video action recognition dataset based on 13320 actions belonging to 101 action classes.

Original: For this dataset we consider a framebased action recognition task.
Revised: For this dataset we consider a frame-based action recognition task.

Original: The images are resized to 224 × 224 and 227 × 227 to ﬁt the input layers of the VGG and AlexNet architectures, respectively.
Revised: The images are resized to 224 ×and 227 × 227 to ﬁt the input layers of the VGG and AlexNet architectures, respectively.

Original: We present results on four architectures.
Revised: We present results on four architectures.

Original: Relatively shallow architectures are trained from scratch, and deeper popular architectures are ﬁne-tuned after being pre-trained on ImageNet [Deng et al.
Revised: Relatively shallow architectures are trained from scratch, and deeper popular architectures are ﬁne-tuned after being pre-trained on ImageNet [Deng et al.

Original: Models trained from scratch: We train the CNN-2 architecture used in Morerio et al.
Revised: Models trained from scratch: We train the CNN-2 architecture used in Morerio et al.

Original: (2017), the state-of-the-art dropout variant, for comparison purposes.
Revised: (2017), the state-of-the-art dropout variant, for comparison purposes.

Original: This architecture consists of three convolutional and two fully-connected layers.
Revised: This architecture consists of three convolutional and two fully-connected layers (see Appendix).

Original: We train this network from scratch for 100 K iterations on the datasets: Cifar-10, Cifar-100 and Caltech-256.
Revised: We train this network from scratch for 100 K iterations on the datasets: Cifar-10, Cifar-100 and Caltech-256.

Original: We use mini-batches of 100 images and ﬁx the learning rate to be 10 − 3 , decreasing to 10 − 4 after 25 K iterations.
Revised: We use minibatches of 100 images and ﬁx the learning rate to be 10 − 3 , decreasing to 10 − 4 after 25 K iterations.

Original: Fine-tuned models:
Revised: Fine-tuned models:

Original: We ﬁne-tune the commonly used architectures: AlexNet [Krizhevsky et al.
Revised: We ﬁne-tune the commonly used architectures: AlexNet [Krizhevsky et al.

Original: (2012)], VGG16 and VGG19 [Simonyan & Zisserman (2014)] pre-trained on ImageNet.
Revised: (2012)], VGG16 and VGG19 [Simonyan & Zisserman (2014)] pre-trained on ImageNet.

Original: We ﬁne-tune the models for a frame by frame action recognition task on UCF101.
Revised: We ﬁnetune the models for a frame by frame action recognition task on UCF101.

Original: The learning rate is ﬁxed to 10 − 3 for all the processes.
Revised: The learning rate is ﬁxed to 10 − 3 for all the processes.

Original: We ﬁne-tune AlexNet for 5 K while VGG16 and VGG19 for 30 K iterations.
Revised: We ﬁne-tune AlexNet for 5 K while VGG16 and VGG19 for 30 K iterations.

Original: We use a batch size of 128 and 50 images for AlexNet and VGG16/19, respectively.
Revised: We use a batch size of 128 and 50 images for AlexNet and VGG16/19, respectively.

Original: 4. S ETUP AND R ESULTS : G ENERALIZATION
Revised: 4. S ETUP AND R ESULTS : G ENERALIZATION

Original: In this section we compare the performance of ED to that of No Dropout, StandardDropout, and
Revised: In this section we compare the performance of Excitation Dropout to that of No Dropout, Standard

Original:  Curriculum Dropout [Morerio et al.
Revised: Dropout, and Curriculum Dropout [Morerio et al.

Original: We train a CNN-2 model from scratch on the datasets: Cifar10, Cifar100, Caltech256.
Revised: We train a CNN-2 model from scratch on the datasets: Cifar10, Cifar100, Caltech256.

Original: Fig.
Revised: Fig.

Original: 3 depicts the test accuracies over training iterations for the three datasets averaged over ﬁve trained models.
Revised: 3 depicts the test accuracies over training iterations for the three datasets averaged over ﬁve trained models.

Original: After convergence, ED demonstrates a signiﬁcant improvement in performance compared to other methods.
Revised: After convergence, Excitation Dropout demonstrates a signiﬁcant improvement in performance compared to other methods.

Original: We hypothesize that ED takes longer to converge due to the additional loop (Steps 2-4 in Fig. 1) introduced in the learning process, and due
Revised: We [Table] [Table] Entropy of

Original: [Table] 
Revised: [Table] [Table]

Revised: Conservative Filters

Original: [Table]
Revised: [Table]

Revised: C a lt ec h # Neurons ON

Original: [Table]
Revised: [Table]

Original: to the learning of the alternative paths.
Revised: Conservative Filters [Table] [Table] [Table]

Revised: U

Revised: CF # Neurons ON

Revised: [Table]

Revised: Conservative Filters

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: over ﬁve trained models for each of the datasets: Cifar10,

Revised: Cifar100, Caltech256 and UCF101 ( σ in brackets).

Revised: Excitation Dropout consistently produces more neurons with non-zero activations, has a more spread saliency map leading to a lower saliency peak, has a higher entropy of both activations and saliency, and has a lower number of conservative ﬁlters; all reﬂecting an improved utilization of the network neurons using Excitation Dropout.

Revised: hypothesize that Excitation Dropout takes longer to converge due to the additional loop (Steps 2-4 in

Revised: Fig. 1) introduced in the learning process, and due to the learning of the alternative paths.

Original: We note that ED, during training, uses a different binary mask for each image in a minibatch, while in Standard Dropout, one random mask is employed per minibatch.
Revised: We note that Excitation Dropout, during training, uses a different binary mask for each image in a minibatch, while in Standard Dropout, one random mask is employed per minibatch.

Original: To prove that  the actual boost in accuracy with ED is notprovided by the choice of speciﬁc masks, we add a comparison with Standard Dropout having a different random mask for each image.
Revised: To prove that it is precisely the fact that masks reﬂective of the particular input give rise to a boost in accuracy, and not the fact that different masks are used for different images, we add a comparison with Standard Dropout having a different random mask for each image.

Original: We refer to this accuracy as ‘Standard Dropout + Mask/Img’ in the plots.
Revised: We refer to this accuracy as ‘Standard Dropout + Mask/Img’ in the plots.

Original: As expected, the latter approach is comparable to Standard Dropout in performance
Revised: As expected, the latter approach is comparable to Standard Dropout in performance.

Original: Next, we evaluate the effectiveness of ED on popular network architectures that employ dropout layers: AlexNet, VGG16, VGG19.
Revised: Next, we evaluate the effectiveness of Excitation Dropout on popular network architectures that employ dropout layers: AlexNet, VGG16, VGG19.

Original: This is done by ﬁne-tuning on the video recognition dataset UCF101.
Revised: This is done by ﬁne-tuning on the video recognition dataset UCF101.

Original: Fig.
Revised: Fig.

Original: 3 shows superior ED performance on AlexNet ﬁne-tuned on UCF101.
Revised: 3 shows superior Excitation Dropout performance on AlexNet ﬁne-tuned on UCF101.

Original: Tablepresents more comparative results on other deep architectures by reporting the accuracy after convergence.
Revised: Table 1 presents more comparative results on other deep architectures by reporting the accuracy after convergence.

Original: Again, ED demonstrates higher generalizability on the test data for all architectures.
Revised: Again, Excitation Dropout demonstrates higher generalizability on the test data for all architectures.

Revised: [Figure]

Revised: Excitation Dropout shows more robustness when more neurons are switched off.

Revised: This is demonstrated through its ability to recover more of the saliency map even when a high percentage of the most salient neurons is dropped-out.

Revised: This ability reﬂects the alternative learnt paths.

Revised: Histograms of the leftmost and rightmost saliency maps are presented to demonstrate that Excitation Dropout has a wider range of saliency values.

Original: For fair comparison, we set p = 0 .
Revised: For fair comparison, we set p = 0 .

Original: 5 for Standard Dropout and P = 0 .
Revised: 5 for Standard Dropout and P = 0 .

Original: 5 for the base retaining probability of ED in all experiments 1 .
Revised: 5 for the base retaining probability of Excitation Dropout in all experiments 1 .

Original: We perform dropout in the ﬁrst fully-connected layer of the[Figure] networks ( fc1 for CNN-2 and fc6 for AlexNet and VGGs) for Standard, Curriculum, andExcitation
Revised: We perform dropout in the ﬁrst fully-connected layer of the networks ( fc1 for CNN-2 and fc6 for AlexNet and VGGs) for Standard, Curriculum, and

Original:  Dropout.
Revised: Excitation Dropout.

Original: For Curriculum Dropout we ﬁx the parameter γ to 5 ∗ 10 − 4 as in Morerio et al.
Revised: For Curriculum Dropout we ﬁx the parameter γ to 5 ∗ 10 − 4 as in Morerio et al.

Original: 4. S ETUP AND R ESULTS : U TILIZATION OF N ETWORK N EURONS
Revised: 4. S ETUP AND R ESULTS : U TILIZATION OF N ETWORK N EURONS

Original: In this section we examine how ED expands the network’s utilization of neurons through the ability of re-wiring or having multiple paths for a certain task.
Revised: In this section we examine how Excitation Dropout expands the network’s utilization of neurons through the learnt alternative paths for a certain task.

Original: Mittal et al.
Revised: Mittal et al.

Original: (2018) introduced scoring functions to rank the ﬁlters in speciﬁc network layers including the average percentage of zero activations , a metric to count how many neurons have zero activations, and the entropy of activations , a metric to measure how much information is contained in the neurons of a layer.
Revised: (2018) introduced scoring functions to rank the ﬁlters in speciﬁc network layers including the average percentage of zero activations , a metric to count how many neurons have zero activations, and the entropy of activations , a metric to measure how much information is contained in the neurons of a layer.

Original: We analogously compute the  entropy of p EB which is higher when the probability distribution is spread out over more neurons in a layer.
Revised: We analogously compute the Neurons ON which is the average number of non-zero activations, the entropy of p EB which is higher when the probability distribution is spread out over more neurons in a layer.

Original: We also compute the peak p EB which is expected to be lower on a more spread distribution.
Revised: We also compute the peak p EB which is expected to be lower on a more spread distribution.

Original: Moreover, Ma et al.
Revised: Moreover, Ma et al.

Original: (2017) introduced conservative ﬁlters : ﬁlters whose parameters do not change signiﬁcantly during training.
Revised: (2017) introduced conservative ﬁlters : ﬁlters whose parameters do not change signiﬁcantly during training.

Original: Conservative ﬁlters reduce the effective number of parameters in a CNN and may limit the CNNs modeling capacity for the target task.
Revised: Conservative ﬁlters reduce the effective number of parameters in a CNN and may limit the CNNs modeling capacity for the target task.

Original: A conservative ﬁlter is a ﬁlter k in layer n whose weights have changed by ∆ kn
Revised: A conservative ﬁlter is a ﬁlter k in layer n whose weights have changed by ∆ kn

Original: = (cid:107)
Revised: = (cid:107)

Original: ˆ w kn − w kn
Revised: ˆ w kn − w kn

Original: (cid:107) , where ∆ kn is less than a threshold ∆ .
Revised: (cid:107) , where ∆ kn is less than a threshold ∆ (empirically set).

Original: We evaluate the presented metrics for Excitation Dropout and compare against Standard and Curriculum Dropout in Table 2.
Revised: We evaluate the presented metrics for Excitation Dropout and compare against Standard and Curriculum Dropout in Table 2.

Original: This is done on the same datasets and architectures considered in
Revised: This is done on the same datasets and architectures considered in

Original: Sec. 4.2.
Revised: Sec. 4.2.

Original: All metrics are computed for the ﬁrst fully-connected layer of the CNN-2 and VGG nets consisting of 2048 and 4096 neurons, respectively.
Revised: All metrics are computed for the ﬁrst fully-connected layer of the CNN-2 and VGG nets consisting of 2048 and 4096 neurons, respectively.

Original: We compute each metric over the test set of each dataset.
Revised: We compute each metric over the test set of each dataset.

Original: Excitation Dropout consistently outperforms Standard and Curriculum Dropout in all the metrics over all datasets.
Revised: Excitation Dropout consistently outperforms Standard and Curriculum Dropout in all the metrics over all datasets.

Original: ED shows a higher number of active neurons, a higher entropy over activations, a probability distribution p EB that is more spread (higher entropy over p EB ) among the neurons of the layer, leading to a lower peak probability of p EB .
Revised: Excitation Dropout shows a higher number of active neurons, a higher entropy over activations, a probability distribution p EB that is more spread (higher entropy over p EB ) among the neurons of the layer, leading to a lower peak probability of p EB and therefore less specialized neurons.

Revised: Averaging models having less specialized neurons results in higher robustness to information loss.

Original: We also observe a signiﬁcantly smaller number of conservative ﬁlters when using ED.
Revised: We also observe a signiﬁcantly smaller number of conservative ﬁlters when using Excitation Dropout.

Original: Fewer ﬁlters remain unchanged, i.e. do not sufﬁciently learn anything far from the random initialization.
Revised: Fewer ﬁlters remain unchanged, i.e. do not sufﬁciently learn anything far from the random initialization.

Original: These results show that the models trained with ED were trained to be more informative, i.e. the contribution for the ﬁnal classiﬁcation task is provided by a higher number of neurons in the network, reﬂecting the alternative learnt paths. 
Revised: These results show that the models trained with Excitation Dropout were trained to be more informative, i.e. the contribution for the ﬁnal classiﬁcation task is provided by a higher number of neurons in the network, reﬂecting the alternative learnt paths. An analysis of such metrics over the training iterations is presented in the Appendix.

Original: 4. S ETUP AND R ESULTS : R ESILIENCE TO C OMPRESSION
Revised: 4. S ETUP AND R ESULTS : R ESILIENCE TO C OMPRESSION

Original: In this section, we simulate ‘Brain Damage’ by dropping out neurons at test time.
Revised: In this section, we simulate ‘Brain Damage’ by dropping out neurons at test time.

Original: Fig.
Revised: Fig.

Original: 4 demonstrates  a network re-wiring itself in order to capture the evidence of the class
Revised: 4 demonstrates how a network utilizes the learnt alternative paths to capture the evidence of the class

Original: HorseRiding in a video frame of the UCF101 dataset.
Revised: HorseRiding in a video frame of the UCF101 dataset.

Original: Given a VGG16 model ﬁne-tuned with Excitation, Curriculum, Standard, and No Dropout at the fc 6 layer, we show the excitation saliency map obtained at the conv5layer as we drop out a ﬁxed number of the most relevant neurons from the same layer dropout is performed upon during training.
Revised: Given a VGG16 model ﬁne-tuned with Excitation, Curriculum, Standard, and No Dropout at the fc 6 layer, we show the excitation saliency map obtained at the conv5-1 layer as we drop out a ﬁxed number of the most relevant neurons from the same layer dropout is performed upon during training.

Original: A neuron is considered to be more relevant if it has a higher p EB .
Revised: A neuron is considered to be more relevant if it has a higher p EB .

Original: In the ﬁrst column of  Fig. 4, the original saliency maps for the different models are shown.
Revised: In the ﬁrst column of frames of Fig. 4, the original saliency maps for the different models are shown.

Original: As already highlighted in Table 2, the original saliency map obtained from the model trained with ED is more spread as compared to that of the other schemes, which present more pronounced red peaks.
Revised: As already highlighted in Table 2, the original saliency map obtained from the model trained with Excitation Dropout is more spread as compared to that of the other schemes, which present more pronounced red peaks.

Original: In the following columns(from 2nd to 6th) of Fig. 4, we present the saliency maps the model is able to restore when the 100 , 200 , 300 , 400 , 500 most relevant neurons are dropped-out.
Revised: In the following columns of Fig. 4, we present the saliency maps the model is able to restore when the 100 , 200 , 300 , 400 , 500 most relevant neurons are dropped-out.

Original: Despite the increasing number of relevant neurons being dropped-out, ED is capable of restoring more of the saliency map contributing to HorseRiding .
Revised: Despite the increasing number of relevant neurons being dropped-out, Excitation Dropout is capable of restoring more of the saliency map contributing to HorseRiding .

Original: This means that the network with ED was trained to ﬁnd alternative paths which belong to the same HorseRiding -relevant cues of the image.
Revised: This means that the network with Excitation Dropout was trained to ﬁnd alternative paths which belong to the same HorseRiding -relevant cues of the image.

Original: Despite the fact that we are considering the worst-case scenario, where we are switching off the most relevant neurons at test time, ED shows most robustness.
Revised: Despite the fact that we are considering the worst-case scenario, where we are switching off the most relevant neurons at test time, Excitation Dropout shows most robustness.

Original: More examples are presented in the Appendix.
Revised: More examples are presented in the Appendix.

Original: While Fig.
Revised: While Fig.

Original: 4 visualizes one example qualitatively, Fig.
Revised: 4 visualizes one example qualitatively, Fig.

Original: 5 presents a complete quantitative analysis on the entire test set after training is complete.
Revised: 5 presents a complete quantitative analysis on the entire test set after training is complete.

Original: We study how the predicted ground-truth (GT) probability changes as more neurons are dropped-out at test time.
Revised: We study how the predicted ground-truth (GT) probability changes as more neurons are dropped-out at test time.

Original: On the left we present the worst case when the neurons dropped are the most relevant to the prediction.
Revised: On the left we present the worst case when the neurons dropped are the most relevant to the prediction.

Original: The horizontal axis in the graph represents p c , where 0 ≤ p c ≤ 1 is the cumulative sum of p EB ofthe most ‘important’ neurons which will be switched off.
Revised: The horizontal axis in the graph represents p c , where 0 ≤ p c ≤ 1 is the cumulative sum of p EB of neurons which will be switched off starting from the most ‘important’.

Original: The analysis is performed for p c = { 0 , 0 .
Revised: The analysis is performed for p c = { 0 , 0 .

Original: In the center we present an analogous analysis dropping the ‘least’ relevant neurons.
Revised: In the center, we present an analogous analysis starting to drop from the ‘least’ relevant neurons.

Original: On the right, we present the random case (more realistic) when k neurons ( k = 0 , 128 , 256 , . . . , 4096 ) are randomly switched off.
Revised: On the right, we present the random case (more realistic) when k neurons ( k = 0 , 128 , 256 , . . . , 4096 ) are randomly switched off.

Original: As we drop more neurons, ED (purple curves) is capable of maintaining a much less steep decline of GT probability, indicating more robustness against network compression.
Revised: As we drop more neurons, Excitation Dropout (purple curves) is capable of maintaining a much less steep decline of GT probability, indicating more robustness against net- work compression.

Original: Cifar10, Cifar100, and Caltech256 show similar behavior (see analogous plots in the Appendix).
Revised: Cifar10, Cifar100, and Caltech256 show similar behavior (see analogous plots in the Appendix).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a new regularization scheme that encourages the learning of alternative paths(re-wiring) in a neural network by deliberately paralyzing high-saliency neurons that contribute more to a network’s prediction during training.
Revised: We propose a new regularization scheme that encourages the learning of alternative paths in a neural network by deliberately paralyzing high-saliency neurons that contribute more to a network’s prediction during training.

Original: In experiments on four image/video recognition datasets, and on different architectures, we demonstrate that our approach yields better generalization on unseen data, higher utilization of network neurons, and higher resilience to network compression.
Revised: In experiments on four image/video recognition datasets, and on different architectures, we demonstrate that our approach yields better generalization on unseen data, higher utilization of network neurons and higher resilience to network compression

Original: APPENDIX
Revised: APPENDIX

Original: Model complexity.
Revised: CNN-2 architecture. Table 3 shows the details of the CNN-2 architecture adopted for Cifar10, Cifar100, and Caltech256 experiments. The size of the softmax layer depends upon the number of classes for each dataset.

Revised: [Table]

Revised: Least vs. most relevant neurons.

Revised: Popular dropout methods ( e.g. Adaptive Dropout Ba & Frey (2013)) drop useless neurons with low activations during training.

Revised: In this work we motivate and demonstrate that dropping neurons based on their Excitation Backprop (EB) probability has added beneﬁts.

Revised: Please note that we are not simply considering neuron activation.

Revised: We demonstrate the performance of Excitation Dropout compared to the variant Adaptive Dropout [Ba & Frey (2013)] in Table 4.

Revised: In essence, Adaptive and Excitation Dropout are opposites by dropping the least and most important neurons, respectively.

Original:  Excitation Dropout(ED) consistently outperforms Standard Dropout (SD), with zero increase in test-time computational complexity.
Revised: [Table] Model complexity. Excitation Dropout consistently outperforms Standard Dropout (SD), with zero increase in test-time computational complexity.

Original: In training, there is a moderate increase in computation: in the worst case, ED will take double (same O-notation complexity) the training time of SD.
Revised: In training, there is a moderate increase in computation: in the worst case, Excitation Dropout will take double (same O-notation complexity) the training time of SD.

Original: This will happen when the utilized ED maps are at the ﬁrst layer of the network.
Revised: This will happen when the utilized Excitation Dropout maps are at the ﬁrst layer of the network.

Original: If a middle layer map is used, ED requires an additional partial forward-backward pass.
Revised: If a middle layer map is used, Excitation Dropout requires an additional partial forward-backward pass.

Original: We use maps of fc layers close to the end of the network to reduce this overhead.
Revised: We use maps of fc layers close to the end of the network to reduce this overhead.

Original: Table 3 presents a run-time analysis for the two main architectures used in this work and compares it to that of Standard Dropout.
Revised: Table 5 presents a run-time analysis for the two main architectures used in this work and compares it to that of Standard Dropout.

Original: [Table]
Revised: [Table]

Original: Sensitivity analysis.
Revised: Sensitivity analysis.

Original: In this section we present a sensitivity analysis over the dropout rate hyperparameter.
Revised: In this section we present a sensitivity analysis over the dropout rate hyperparameter.

Original: In our work, we implement ED with a base retaining probability P [Table] and we compare that to standard dropout with a retaining probability p , where P = p .
Revised: In our work, we implement Excitation Dropout with a base retaining probability P , and we compare that to standard dropout with a retaining probability p , where P = p .

Original: If ED produced a uniform probability distribution over the desired layer, then every node would have a retain probability equal to the base probability P .
Revised: If Excitation Dropout produced a uniform probability distribution over the desired layer, then every node would have a retain probability equal to the base probability P .

Original: For completeness, we add a sensitivity analysis of the parameters p and P in Table4.
Revised: For completeness, we add a sensitivity analysis of the parameters p and P in Table

Original: [Table]
Revised: [Table]

Original: Least vs. most relevant neurons.
Revised: Excitation Dropout in Convolutional Layers. Excitation Dropout is a generic formulation that can be applied to any neural network layer. For a convolutional layer, a generic convolutional activation map is in the form of [ w, h, N ] , where N is the number of feature maps while w and h are the spatial dimensions. To apply Excitation Dropout to a convolutional layer, ﬁrst p EB is computed for each feature map N as the sum of p EB across spatial locations w and h .

Original: Popular dropout methods ( e.g. Adaptive Dropout Ba & Frey (2013)) drop useless neurons with low activations during training.
Revised: Speciﬁc 2D feature maps are then dropped-out following Eqn. 3. We test Excitation Dropout at conv 3 of the CNN-2 architecture and obtain the following accuracy results for Cifar10: No Dropout 76 . 91% ; Excitation Dropout at conv 3 78 .

Original: In this work we motivate and demonstrate that dropping neurons based on their Excitation Backprop (EB) probability has added beneﬁts.
Revised: 01% ; Excitation Dropout at fc 1 81 . Again, we observe an improvement respect to No Dropout, but consistent with the literature [Hinton et al. ; Srivastava et al.

Original: Please note that we are not simply considering neuron activation.
Revised: , the improvement is not as large as using dropout in fully connected layers.

Original: We demonstrate the performance of ED compared to the variant Adaptive Dropout [Ba & Frey (2013)] in Table 5.
Revised: Validation on Multiple Architectures. In this section we report additional results for Caltech and Cifar10 datasets considering different architectures. Table 7 reports the results of ﬁne-tuning the deep architectures AlexNet, VGG16 and VGG19 on Caltech256 for the task of image classiﬁcation. Finally, we adopt the WideResNet (WRN-28-10) architecture, a ResNet style architecture which combines batch normalization and dropout regularization techniques.

Original: In essence, Adaptive and Excitation Dropout are opposites by dropping the least and most important neurons, respectively
Revised: WideResNet is used to obtain state-of-the-art results on Cifar10. We replace the Standard Dropout layer in the network with Excitation Dropout obtaining 3.88% test error on Cifar10 (vs. 4.17% Zagoruyko et al.s published result, BMVC16). Therefore, Excitation Dropout gives state-of-the-art result on Cifar10.

Original: [Table]
Revised: [Table]

Revised: , averaged over ﬁve trained models.

Original: Additional Visualizations.
Revised: Additional Visualizations.

Original: In this section, we present more visualizations similar to that of Fig.in the main manuscript.
Revised: In this section, we present more visualizations similar to that of Fig.in the main manuscript.

Original: Visualizations for a VGG16 network ﬁne-tuned on UCF101 for the actions:
Revised: Visualizations for a VGG16 network ﬁne-tuned on UCF101 for the actions:

Original: PlayingFlute , PlayingSitar , and GolfSwing are presented in Fig.s 6,7,8, respectively.
Revised: PlayingFlute , PlayingSitar , and GolfSwing are presented in Fig.s 6,7,8, respectively.

Original: Every column displays the saliency map over the same video frame of an action while incrementally switching off the most k relevant neurons ( k = 0 , 100 , 200 , . .
Revised: Every column displays the saliency map over the same video frame of an action while incrementally switching off the most k relevant neurons ( k = 0 , 100 , 200 , . .

Original: Excitation Dropout is more robust over higher number of switched off neurons.
Revised: Excitation Dropout is more robust over higher number of switched off neurons.

Original: This is demonstrated through its ability to recover the saliency map even when a high percentage of the most salient neurons is dropped-out.
Revised: This is demonstrated through its ability to recover the saliency map even when a high percentage of the most salient neurons is dropped-out.

Original: This is done through the alternative learnt paths, which are reﬂected in the higher number of non-zero activations that remain after dropout compared to other dropout strategies. 
Revised: This is done through the alternative learnt paths, which are reﬂected in the higher number of non-zero activations that remain after dropout compared to other dropout strategies. [Figure]

Original: Additional Plots.
Revised: Additional Plots.

Original: In this section we present the fc compression results for three additional datasets:
Revised: In this section we present the fc compression results for three additional datasets:

Original: Cifar10, Cifar100, and Caltech256 (Fig. 9), using CNN-2.
Revised: Cifar10, Cifar100, and Caltech256 (Fig. 9), using CNN-2.

Original: We study how the predicted ground-truth (GT) probability changes as more neurons are dropped-out at test time.
Revised: We study how the predicted ground-truth (GT) probability changes as more neurons are dropped-out at test time.

Original: On the left we present the worst case when the neurons dropped are the most relevant to the prediction.
Revised: On the left we present the worst case when the neurons dropped are the most relevant to the prediction.

Original: The horizontal axis in the graph represents p c , where 0 ≤ p c ≤ 1 is the cumulative sum of p EB ofthe most ‘important’ neurons which will be switched off.
Revised: The horizontal axis in the graph represents p c , where 0 ≤ p c ≤ 1 is the cumulative sum of p EB of neurons which will be switched off starting from the most ‘important’.

Original: The analysis is performed for p c = { 0 , 0 .
Revised: The analysis is performed for p c = { 0 , 0 .

Original: In the center, we present an analogous analysis dropping the ‘least’ relevant neurons.
Revised: In the center, we present an analogous analysis starting to drop from the ‘least’ relevant neurons.

Original: On the right, we present the random case (more realistic) when k neurons ( k = 0 , 128 , 256 , . . . , 2048 ) are randomly switched off.
Revised: On the right, we present the random case (more realistic) when k neurons ( k = 0 , 128 , 256 , . . . , 2048 ) are randomly switched off.

Original: As we drop more neurons, Excitation Dropout (purple curves) is capable of maintaining a much less steep decline of GT probability. 
Revised: As we drop more neurons, Excitation Dropout (purple curves) is capable of maintaining a much less steep decline of GT probability. Metric Analysis During Training.

Original: [Figure]
Revised: In this section we report an extended analysis of the metrics: # Neurons ON , Peak p EB , Entropy of Activations , and Entropy of p EB during training. Excitation Dropout shows a higher number of active neurons, a higher entropy over activations, a probability distribution p EB that is more spread (higher entropy over p EB ) among the neurons of the layer, leading to a lower peak probability of p EB and therefore less specialized neurons. These results are observed to have consistent trends over all training iterations for all datasets considered (see

Revised: Fig.s 10, 11, 12, and 13)

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]