Original: A DAPTIVE D ROPOUT WITH R ADEMACHER C OMPLEXITY R EGULARIZATION
Revised: A DAPTIVE D ROPOUT WITH R ADEMACHER C OMPLEXITY R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound.
Revised: We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound.

Original: The state-ofthe-art deep learning algorithms impose dropout strategy to prevent feature coadaptation.
Revised: The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation.

Original: However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space.
Revised: However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space.

Original: In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefﬁcient matrices.
Revised: In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefﬁcient matrices.

Original: Subsequently, we impose this bound as a regularizer and provide a theoretical justiﬁed way to tradeoff between model complexity and representation power.
Revised: Subsequently, we impose this bound as a regularizer and provide a theoretical justiﬁed way to trade-off between model complexity and representation power.

Original: Therefore, the dropout rates and the empirical loss are uniﬁed into the same objective function, which is then optimized using the block coordinate descent algorithm.
Revised: Therefore, the dropout rates and the empirical loss are uniﬁed into the same objective function, which is then optimized using the block coordinate descent algorithm.

Original: We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.
Revised: We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.

Original: Experiments on the task of image and document classiﬁcation also show our method achieves better performance compared to the state-of-the-art dropout algorithms.
Revised: Experiments on the task of image and document classiﬁcation also show our method achieves better performance compared to the state-of-the-art dropout algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dropout training (Srivastava et al., 2014) has been proposed to regularize deep neural networks for classiﬁcation tasks.
Revised: Dropout training (Srivastava et al., 2014) has been proposed to regularize deep neural networks for classiﬁcation tasks.

Original: It has been shown to work well in reducing co-adaptation of neurons—and hence, preventing model overﬁtting.
Revised: It has been shown to work well in reducing co-adaptation of neurons—and hence, preventing model overﬁtting.

Original: The idea of dropout is to stochastically set a neuron’s output to zero according to Bernoulli random variables.
Revised: The idea of dropout is to stochastically set a neuron’s output to zero according to Bernoulli random variables.

Original: It has been a crucial component in the winning solution to visual object recognition on ImageNet (Krizhevsky et al., 2012).
Revised: It has been a crucial component in the winning solution to visual object recognition on ImageNet (Krizhevsky et al., 2012).

Original: Ever since, there have been many follow-ups on novel learning algorithms (Goodfellow et al., 2013; Baldi & Sadowski, 2013), regularization techniques (Wager et al., 2013), and fast approximations (Wang & Manning, 2013).
Revised: Ever since, there have been many follow-ups on novel learning algorithms (Goodfellow et al., 2013; Baldi & Sadowski, 2013), regularization techniques (Wager et al., 2013), and fast approximations (Wang & Manning, 2013).

Original: However, the classical dropout model has two limitations.
Revised: However, the classical dropout model has two limitations.

Original: First, the model requires to specify the retain rates, i.e., the probabilities of keeping a neuron’s output, a priori to model training.
Revised: First, the model requires to specify the retain rates, i.e., the probabilities of keeping a neuron’s output, a priori to model training.

Original: It is often not clear how to choose the retain rates in an optimal way.
Revised: It is often not clear how to choose the retain rates in an optimal way.

Original: They are usually set via grid-search over hyper-parameter space or simply according to some rule-of-thumb, and kept consistent throughout the training process thereafter.
Revised: They are usually set via grid-search over hyper-parameter space or simply according to some rule-of-thumb, and kept consistent throughout the training process thereafter.

Original: Another limitation is that all neurons in the same layer share the same retain rate.
Revised: Another limitation is that all neurons in the same layer share the same retain rate.

Original: This exponentially reduces the search space of hyper-parameter optimization.
Revised: This exponentially reduces the search space of hyper-parameter optimization.

Original: For example, Srivastava et al.
Revised: For example, Srivastava et al.

Original: (2014) use a ﬁxed retain probability throughout training for all dropout variables in each layer.
Revised: (2014) use a ﬁxed retain probability throughout training for all dropout variables in each layer.

Original: In this paper, we propose a novel regularizer based on the Rademacher complexity of a neural network (Shalev-Shwartz & Ben-David, 2014).
Revised: In this paper, we propose a novel regularizer based on the Rademacher complexity of a neural network (Shalev-Shwartz & Ben-David, 2014).

Original: Without loss of generality, we use multilayer perceptron with dropout as our example and prove its Rademacher complexity is bounded by a term related to the dropout probabilities.
Revised: Without loss of generality, we use multilayer perceptron with dropout as our example and prove its Rademacher complexity is bounded by a term related to the dropout probabilities.

Original: This enables us to explicitly incorporate the model complexity term as a regularizer into the objective function.
Revised: This enables us to explicitly incorporate the model complexity term as a regularizer into the objective function.

Original: This Rademacher complexity bound regularizer provides us a lot of ﬂexibility and advantage in modeling and optimization.
Revised: This Rademacher complexity bound regularizer provides us a lot of ﬂexibility and advantage in modeling and optimization.

Original: First, it combines the model complexity and the loss function in an uniﬁed objective.
Revised: First, it combines the model complexity and the loss function in an uniﬁed objective.

Original: This offers a viable way to trade-off the model complexity and representation power through the regularizer weighting coefﬁcient.
Revised: This offers a viable way to trade-off the model complexity and representation power through the regularizer weighting coefﬁcient.

Original: Second, since this bound is a function of dropout probabilities, we are able to incorporate them explictly into the computation graph of the optimization procedure.
Revised: Second, since this bound is a function of dropout probabilities, we are able to incorporate them explictly into the computation graph of the optimization procedure.

Original: We can then adaptively optimize the objective and adjust the dropout probabilities throughout training in a way similar to ridge regression and the lasso (Hastie et al., 2009).
Revised: We can then adaptively optimize the objective and adjust the dropout probabilities throughout training in a way similar to ridge regression and the lasso (Hastie et al., 2009).

Original: Third, our proposed regularizer assumes a neuron-wise dropout manner and models different neurons to have different retain rates during the optimization.
Revised: Third, our proposed regularizer assumes a neuron-wise dropout manner and models different neurons to have different retain rates during the optimization.

Original: Our empirical results demonstrate interesting trend on the changes in histograms of dropout probabilities for both hidden and input layers.
Revised: Our empirical results demonstrate interesting trend on the changes in histograms of dropout probabilities for both hidden and input layers.

Original: We also discover that the distribution over retain rates upon model convergence reveals meaningful pattern on the input features.
Revised: We also discover that the distribution over retain rates upon model convergence reveals meaningful pattern on the input features.

Original: To the best of our knowledge, this is the ﬁrst ever effort of using the Rademacher complexity bound to adaptively adjust the dropout probabilities for the neural networks.
Revised: To the best of our knowledge, this is the ﬁrst ever effort of using the Rademacher complexity bound to adaptively adjust the dropout probabilities for the neural networks.

Original: We organize the rest of the paper as following.
Revised: We organize the rest of the paper as following.

Original: Section 2 reviews some past approaches well aligned with our motivation, and highlight some major difference to our proposed approach.
Revised: Section 2 reviews some past approaches well aligned with our motivation, and highlight some major difference to our proposed approach.

Original: We subsequently detail our proposed approach in Section 3.
Revised: We subsequently detail our proposed approach in Section 3.

Original: In Section 4, we present our thorough empirical evaluations on the task of image and document classiﬁcation on several benchmark datasets.
Revised: In Section 4, we present our thorough empirical evaluations on the task of image and document classiﬁcation on several benchmark datasets.

Original: Finally, Section 5 concludes this paper and summarizes some possible future research ideas.
Revised: Finally, Section 5 concludes this paper and summarizes some possible future research ideas.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: There are several prior works well aligned with our motivation and addressing similar problems, but signiﬁcantly different from our method.
Revised: There are several prior works well aligned with our motivation and addressing similar problems, but signiﬁcantly different from our method.

Original: For example, the standout network (Ba & Frey, 2013) extends dropout network into a complex network structure, by interleaving a binary belief network with a regular deep neural network.
Revised: For example, the standout network (Ba & Frey, 2013) extends dropout network into a complex network structure, by interleaving a binary belief network with a regular deep neural network.

Original: The binary belief network controls the dropout rate for each neuron, backward propagates classiﬁcation error and adaptively adjust according to training data.
Revised: The binary belief network controls the dropout rate for each neuron, backward propagates classiﬁcation error and adaptively adjust according to training data.

Original: Zhuo et al.
Revised: Zhuo et al.

Original: (2015) realize the dropout training via the concept of Bayesian feature noising, which incorporates dimension-speciﬁc or group-speciﬁc noise to adaptively learn the dropout rates.
Revised: (2015) realize the dropout training via the concept of Bayesian feature noising, which incorporates dimension-speciﬁc or group-speciﬁc noise to adaptively learn the dropout rates.

Original: In addition to these approaches, one other family of solution is via the concept of regularizer.
Revised: In addition to these approaches, one other family of solution is via the concept of regularizer.

Original: Wang & Manning (2013) propose fast approximation methods to marginalize the dropout layer and show that the classical dropout can be approximated by a Gaussian distribution.
Revised: Wang & Manning (2013) propose fast approximation methods to marginalize the dropout layer and show that the classical dropout can be approximated by a Gaussian distribution.

Original: Later, Wager et al.
Revised: Later, Wager et al.

Original: (2013) show that the dropout training on generalized linear models can be viewed as a form of adaptive regularization technique.
Revised: (2013) show that the dropout training on generalized linear models can be viewed as a form of adaptive regularization technique.

Original: Gal & Ghahramani (2016) develop a new theoretical framework casting dropout training as approximation to Bayesian inference in deep Gaussian processes.
Revised: Gal & Ghahramani (2016) develop a new theoretical framework casting dropout training as approximation to Bayesian inference in deep Gaussian processes.

Original: It also provides a theoretical justiﬁcation and formulates dropout into a special case of Bayesian regularization.
Revised: It also provides a theoretical justiﬁcation and formulates dropout into a special case of Bayesian regularization.

Original: In the mean time, Maeda (2014) discuss a Bayesian perspective on dropout focusing on the binary variant, and also demonstrate encourage experimental results.
Revised: In the mean time, Maeda (2014) discuss a Bayesian perspective on dropout focusing on the binary variant, and also demonstrate encourage experimental results.

Original: Generalized dropout (Srinivas & Babu, 2016) further uniﬁes the dropout model into a rich family of regularizers and propose a Bayesian approach to update dropout rates.
Revised: Generalized dropout (Srinivas & Babu, 2016) further uniﬁes the dropout model into a rich family of regularizers and propose a Bayesian approach to update dropout rates.

Original: One popular method along with these works is the variational dropout method (Kingma et al., 2015), which provides an elegant interpretation of Gaussian dropout as a special case of Bayesian regularization.
Revised: One popular method along with these works is the variational dropout method (Kingma et al., 2015), which provides an elegant interpretation of Gaussian dropout as a special case of Bayesian regularization.

Original: It also proposes a Bayesian inference method using a local reparameterization technique and translates uncertainty of global parameters into local noise.
Revised: It also proposes a Bayesian inference method using a local reparameterization technique and translates uncertainty of global parameters into local noise.

Original: Hence, it allows inference on the parameterized Bayesian posteriors for dropout rates.
Revised: Hence, it allows inference on the parameterized Bayesian posteriors for dropout rates.

Original: This allows us to adaptively tune individual dropout rates on layer, neuron or even weight level in a Bayesian manner.
Revised: This allows us to adaptively tune individual dropout rates on layer, neuron or even weight level in a Bayesian manner.

Original: Recently,
Revised: Recently,

Original: Molchanov et al.
Revised: Molchanov et al.

Original: (2017) extend the variational dropout method with a tighter approximation which subsequently produce more sparse dropout rates.
Revised: (2017) extend the variational dropout method with a tighter approximation which subsequently produce more sparse dropout rates.

Original: However, these models are fundamentally different than our proposed approach.
Revised: However, these models are fundamentally different than our proposed approach.

Original: They directly operates on the Gaussian approximation of dropout models rather than the canonical multiplicative dropout model, whereas our proposed method directly bounds the model complexity of classical dropout model.
Revised: They directly operates on the Gaussian approximation of dropout models rather than the canonical multiplicative dropout model, whereas our proposed method directly bounds the model complexity of classical dropout model.

Original: Meanwhile, the model complexity and the generalization capability of deep neural networks have been well studied in theoretical perspective.
Revised: Meanwhile, the model complexity and the generalization capability of deep neural networks have been well studied in theoretical perspective.

Original: Wan et al.
Revised: Wan et al.

Original: (2013) prove the generalization bound for the
Revised: (2013) prove the generalization bound for the

Original: DropConnect neural networks—a weight-wise variant of dropout model.
Revised: DropConnect neural networks—a weight-wise variant of dropout model.

Original: Later, Gao & Zhou (2016) extend the work and derive a Rademacher complexity bound for deep neural networks with dropout.
Revised: Later, Gao & Zhou (2016) extend the work and derive a Rademacher complexity bound for deep neural networks with dropout.

Original: These works provide a theoretical guarantee and mathematical justiﬁcation on the effectiveness of dropout method in general.
Revised: These works provide a theoretical guarantee and mathematical justiﬁcation on the effectiveness of dropout method in general.

Original: However, they both assume that all input and hidden layers have the same dropout rates.
Revised: However, they both assume that all input and hidden layers have the same dropout rates.

Original: Thus their bound can not be applied to our algorithm.
Revised: Thus their bound can not be applied to our algorithm.

Original:  R EGULARIZATION WITH R ADEMACHER C OMPLEXITY
Revised:  R EGULARIZATION WITH R ADEMACHER C OMPLEXITY

Original: We would like to focus on the classiﬁcation problem and use multilayer perceptron as our example.
Revised: We would like to focus on the classiﬁcation problem and use multilayer perceptron as our example.

Original: However, note that the similar idea could be easily extended to general feedforward networks.
Revised: However, note that the similar idea could be easily extended to general feedforward networks.

Original: Let us assume a labeled dataset S
Revised: Let us assume a labeled dataset S

Original: = { ( x i , y i )
Revised: = { ( x i , y i )

Original: | i ∈ { 1 , 2 , . . . , n } , x i ∈ R d , y i ∈ { 0 , 1 } k } , where x i is the feature of the i th sample, y i is the one-hot class label for the i th sample, and k is the number of classes in prediction.
Revised: | i ∈ { 1 , 2 , . . . , n } , x i ∈ R d , y i ∈ { 0 , 1 } k } , where x i is the feature of the i th sample, y i is the one-hot class label for the i th sample, and k is the number of classes in prediction.

Original: Without loss of generality, an L -layer multilayer perceptron with dropout can be modeled as a series of recursive function compositions.
Revised: Without loss of generality, an L -layer multilayer perceptron with dropout can be modeled as a series of recursive function compositions.

Original: Let k l be the number of neurons of the l th layer.
Revised: Let k l be the number of neurons of the l th layer.

Original: In particular, the ﬁrst layer takes sample features as input, i.e., k 0 =
Revised: In particular, the ﬁrst layer takes sample features as input, i.e., k 0 =

Original: d , and the last layer outputs the prediction, i.e., k L = k .
Revised: d , and the last layer outputs the prediction, i.e., k L = k .

Original: We denote W l ∈ R k l − 1 × k l as the linear coefﬁcient matrix from the ( l − 1) th layer to the l th layer, and
Revised: We denote W l ∈ R k l − 1 × k l as the linear coefﬁcient matrix from the ( l − 1) th layer to the l th layer, and

Original: W li be the i th column of W l .
Revised: W li be the i th column of W l .

Original: For dropout, we denote θ l ∈ [0 , 1] k l as the vector of retain rates for the l th layer.
Revised: For dropout, we denote θ l ∈ [0 , 1] k l as the vector of retain rates for the l th layer.

Original: We also deﬁne r l ∈ { 0 , 1 } k l as a binary vector formed by concatenating k l independent
Revised: We also deﬁne r l ∈ { 0 , 1 } k l as a binary vector formed by concatenating k l independent

Original: Bernoulli dropout random variables, i.e., r lj ∼ Bernoulli ( θ lj ) .
Revised: Bernoulli dropout random variables, i.e., r lj ∼ Bernoulli ( θ lj ) .

Original: To simplify our notation, we further refer W : l = { W 1 , . .
Revised: To simplify our notation, we further refer W : l = { W 1 , . .

Original: , W l } , r : l = { r 0 , . . .
Revised: , W l } , r : l = { r 0 , . . .

Original: , r l } , θ : l = { θ 0 , . . .
Revised: , r l } , θ : l = { θ 0 , . . .

Original: , θ l } , W = W : L , r = r :(
Revised: , θ l } , W = W : L , r = r :(

Original: L − 1) , and θ = θ :(
Revised: L − 1) , and θ = θ :(

Original: L − 1) .
Revised: L − 1) .

Original: For an input sample feature vector x ∈ R d , the function before the activation of the j th neuron in the l th layer f lj is
Revised: For an input sample feature vector x ∈ R d , the function before the activation of the j th neuron in the l th layer f lj is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where φ : R → R + is the rectiﬁed linear activation function (Nair & Hinton, 2010, ReLU ).
Revised: where φ : R → R + is the rectiﬁed linear activation function (Nair & Hinton, 2010, ReLU ).

Original: In vector form, if we denote (cid:12) as the Hadamard product, we could write the output of the l th layer as
Revised: In vector form, if we denote (cid:12) as the Hadamard product, we could write the output of the l th layer as

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we also apply Bernoulli dropout to the input layer parameter θ 0 ∈ R d , i.e., f 1 ( x ; W , r 0 ) =
Revised: Without loss of generality, we also apply Bernoulli dropout to the input layer parameter θ 0 ∈ R d , i.e., f 1 ( x ; W , r 0 ) =

Original: ( r 0 (cid:12) x ) W 1 .
Revised: ( r 0 (cid:12) x ) W 1 .

Original: Note that the output of the neural network f L ( x ; W , r ) ∈
Revised: Note that the output of the neural network f L ( x ; W , r ) ∈

Original: R k is a random vector due to the Bernoulli random variables r .
Revised: R k is a random vector due to the Bernoulli random variables r .

Original: We use the expected value of f L ( x ; W , r ) as the deterministic output
Revised: We use the expected value of f L ( x ; W , r ) as the deterministic output

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal predictions are made through a softmax function, and we use the cross-entropy loss as our optimization objective.
Revised: The ﬁnal predictions are made through a softmax function, and we use the cross-entropy loss as our optimization objective.

Original: To simplify our analysis, we follow Wan et al.
Revised: To simplify our analysis, we follow Wan et al.

Original: and reformulate the cross-entropy loss on top of the softmax into a single logistic function
Revised: and reformulate the cross-entropy loss on top of the softmax into a single logistic function

Original: [Equation]
Revised: [Equation]

Original: 3.  R ADEMACHER C OMPLEXITY OF D ROPOUT N EURAL N ETWORK 
Revised: 3. E MPIRICAL R ADEMACHER C OMPLEXITY OF D ROPOUT N EURAL N ETWORK Deﬁnition

Original: Deﬁne loss ◦ f L as the composition ofthe logistic loss function loss and the neural function f L returned from the L th (last) layer, i.e.,
Revised: The empirical Rademacher complexity of function class F with respect to the sample S is [Equation] n (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: [Equation]

Revised: Deﬁne loss ◦ f L as the composition of the logistic loss function loss and the neural function f L returned from the L th (last) layer, i.e., loss ◦ f L = { ( x , y )

Original: x , y ) → loss ( f L ( x ; W , θ ) , y ) } .
Revised: → loss ( f L ( x [Equation] W , θ ) , y ) } .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let X ∈ R n × d be the sample matrix with the i th row x i ∈ R d , p ≥ 1 ,p + 1 q = 1 .
Revised: Let X ∈ R n × d be the sample matrix with the i th row x i ∈ R d , p ≥ 1 ,p + 1 q = 1 .

Original: If the p -norm of every column of W l is bounded by a constant B l , denote
Revised: If the p -norm of every column of W l is bounded by a constant B l , denote

Original: W =
Revised: W =

Original: [Equation]
Revised: [Equation]

Original: W | max j 
Revised: W | max j (cid:107) W lj (cid:107) p

Original: [Equation]
Revised: [Equation]

Original: W lj (cid:107) p ≤ B l , ∀ l ∈ { 1 , 2 , . . .
Revised: l , ∀ l ∈ { 1 , 2 , . . .

Original: , L } (cid:9) , given θ , the empirical Rademacher complexity of the loss for the dropout neural network deﬁned above is bounded by
Revised: , L } (cid:9) , given θ , the empirical Rademacher complexity of the loss for the dropout neural network deﬁned above is bounded by

Original: [Equation]
Revised: [Equation]

Original: where k is the number of classes to predict, θ l is the k l -dimensional vector of Bernoulli parameters for the dropout random variables in the l th layer, σ i s are i.i.d.
Revised: where k is the number of classes to predict, θ l is the k l -dimensional vector of Bernoulli parameters for the dropout random variables in the l th layer, σ i s are i.i.d.

Original: Rademacher random variables, and (cid:107) · (cid:107) max is the matrix max norm deﬁned as (cid:107) A (cid:107) max = max ij |
Revised: Rademacher random variables, and (cid:107) · (cid:107) max is the matrix max norm deﬁned as (cid:107) A (cid:107) max = max ij |

Original: A ij | .
Revised: A ij | .

Revised: [Figure]

Revised: e g u l a r i z e r

Original: Please refer to the appendix for the proof.
Revised: Please refer to the appendix for the proof.

Original: 3. R EGULARIZE D ROPOUT N EURAL N ETWORK WITH R ADEMACHER C OMPLEXITY
Revised: Theorem 3.1 suggests that the empirical Rademacher complexity of the dropout network speciﬁed in this paper is related to several terms in a multiplicative way: i : p -norms of the coefﬁcients:

Revised: [Equation]

Revised: Note that in (Srivastava et al., 2014), 2 norms of the coefﬁcients are already used as regularizers in the experimental comparison

Revised: ii: q -norms of the retain rates θ l iii: sample related metrics: dimension of the sample d , the number of samples n , and maximum entries in the samples

Revised: X iv: the number of classes in the prediction k

Revised: An the extreme case is, if the retain rates θ l for one layer are all zeros, then the upper bound above is tight, since in this case the network is simply doing random guess for predictions.

Revised: Similarly when the coefﬁcients in one layer are all zeros, the bound is also tight.

Revised: In both cases the features from the samples are not even used in the prediction due to either zero retain rates or zero coefﬁcients.

Revised: 3. R EGULARIZE D ROPOUT N EURAL N ETWORK WITH R ADEMACHER C OMPLEXITY

Original: We have shown that the Rademacher complexity of a neural network is bounded by a function of the dropout rates, i.e., Bernoulli parameters θ .
Revised: We have shown that the Rademacher complexity of a neural network is bounded by a function of the dropout rates, i.e., Bernoulli parameters θ .

Original: This makes it possible to unify the dropout rates and the network coefﬁcients W in one objective.
Revised: This makes it possible to unify the dropout rates and the network coefﬁcients W in one objective.

Original: By imposing our upper bound of Rademacher complexity to the loss function as a regularizer, we have
Revised: By imposing our upper bound of Rademacher complexity to the loss function as a regularizer, we have

Original: [Equation]
Revised: [Equation]

Original: where the variable λ ∈ R + is a weighting coefﬁcient to trade off the training loss and the generalization capability.
Revised: where the variable λ ∈ R + is a weighting coefﬁcient to trade off the training loss and the generalization capability.

Original: The empirical loss Loss ( S , f L ( · ; W , θ )) and regularizer function Reg ( S , W , θ ) are deﬁned as
Revised: The empirical loss Loss ( S , f L ( · ; W , θ )) and regularizer function Reg ( S , W , θ ) are deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where W lj is the j th column of W l and θ l is the retain rate vector for the l th layer.
Revised: , where W lj is the j th column of W l and θ l is the retain rate vector for the l th layer.

Original: The variable k is the number of classes to predict and X ∈ R n × d is the sample matrix.
Revised: The variable k is the number of classes to predict and X ∈ R n × d is the sample matrix.

Original: [Figure]

Original: The experiments are evaluated on MNIST dataset with a hidden layer of 128 ReLU units.

Original: We apply dropout on the hidden layer only, and keep the retain rates ﬁxed throughout training.

Original: We optimize the neural network with the empirical loss

Original: Loss ( S , f L ( · ; W , θ )) only, i.e., without any regularizer.

Original: All the values of the Rademacher p = q = 2 regularizer are computed after every epoch in post-hoc manner.

Original: We use minibatch size of 100 , 200 epochs, initial learning rate of 0 . 01 , and decay it by half every 40 epochs.

Original: We plot the samples from last 20 epochs under each settings.

Original: In addition to the Rademacher regularizer Reg ( S , W , θ ) , the empirical loss term Loss ( S , f L ( · ; W , θ )) also depends on the dropout Bernoulli parameters θ .
Revised: In addition to the Rademacher regularizer Reg ( S , W , θ ) , the empirical loss term Loss ( S , f L ( · ; W , θ )) also depends on the dropout Bernoulli parameters θ .

Original: Intuitively, when θ becomes smaller, the loss term Loss ( S , f L ( · ; W , θ )) becomes larger, since the model is less capable to ﬁt the training samples (i.e., less representation power), the empirical Rademacher complexity bound becomes smaller (i.e., more generalizable), and vice versa.
Revised: Intuitively, when θ becomes smaller, the loss term Loss ( S , f L ( · ; W , θ )) becomes larger, since the model is less capable to ﬁt the training samples (i.e., less representation power), the empirical Rademacher complexity bound becomes smaller (i.e., more generalizable), and vice versa.

Original: Figure 1 plots the values of the cross-entropy loss function and empirical Rademacher p = q = 2 regularizer upon model convergence under different ﬁxed settings of retain rates.
Revised: Figure 1 plots the values of the cross-entropy loss function and empirical Rademacher p = q = 2 regularizer upon model convergence under different ﬁxed settings of retain rates.

Original: In the extreme case, when all θ lj become zeros, the model always makes random guess for prediction, leading to a large ﬁtness error Loss ( S , f L ( · ; W , θ )) , and the 
Revised: In the extreme case, when all θ lj become zeros, the model always makes random guess for prediction, leading to a large ﬁtness error Loss ( S , f L ( · ; W , θ )) , and the Rademacher complexity

Original: Rademacher complexity R S ( loss ◦ f L ) approaches 0 .
Revised: R S ( loss ◦ f L ) approaches 0 .

Original: 3. O PTIMIZE D ROPOUT R ATES
Revised: 3. O PTIMIZE D ROPOUT R ATES

Original: We now incorporate the Bernoulli parameters θ into the optimization objective as in Eqn.
Revised: We now incorporate the Bernoulli parameters θ into the optimization objective as in Eqn.

Original: (2), i.e., the objective is a function of both weight coefﬁcient matrices W and retain rate vectors θ .
Revised: (2), i.e., the objective is a function of both weight coefﬁcient matrices W and retain rate vectors θ .

Original: In particular, the model parameters and the dropout out rates are optimized using a block coordinate descent algorithm.
Revised: In particular, the model parameters and the dropout out rates are optimized using a block coordinate descent algorithm.

Original: We start with an initial setting of W and θ , and optimize W and θ in an alternating fashion.
Revised: We start with an initial setting of W and θ , and optimize W and θ in an alternating fashion.

Original: During the optimization of θ , we used the expected value of the  dropout layer to rescale the output from each layer.
Revised: During the optimization of θ , we use the expected value of the Bernoulli dropout variables to rescale the output from each layer.

Original: It signiﬁcantly speeds up the forward propagation process, as we do not needinternal to iteratively sample the dropout variables.
Revised: It signiﬁcantly speeds up the forward propagation process, as we do not need to iteratively sample the dropout variables.

Original: Note that this is an approximation to the true f L ( x ; W , θ ) , however, in practice, we ﬁnd the difference are negligible on each minibatch.

Original: Essentially, it makes the layer output deterministic and the underlying network operates as if without dropout, i.e., similar to the approximation used in (Srivastava et al., 2014) during testing time.
Revised: Essentially, it makes the layer output deterministic and the underlying network operates as if without dropout, i.e., exactly same as the approximation used in (Srivastava et al., 2014) during testing time.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Equation]
Revised: [Table] [Table] [Table]

Original: PARSE V AR D ROP 
Revised: + V AR D ROP 1 .

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Table]

Revised: + S PARSE V AR D ROP

Original: [Table] [Equation]
Revised: [Table] [Table] [Table]

Original:  Rademacher 
Revised: + Rademacher p = q =1 .

Original: [Equation]
Revised: [Table]

Original: ∞
Revised: 0 .+

Revised: Rademacher p = 1 , q = ∞ 1 .

Original: [Table]
Revised: [Table]

Original: + Rademacher [Equation]
Revised: 0 .+ Rademacher p = ∞ , q =1 .

Original: [Table]  We apply our proposed approach with different network architectures, on the task of image and text classiﬁcation using several public available benchmark datasets.
Revised: [Table] [Table] We apply our proposed approach with different network architectures, on the task of image and text classiﬁcation using several public available benchmark datasets.

Original: All hidden neurons and convolutional ﬁlters are rectiﬁed linear units (Nair & Hinton, 2010, ReLU ).
Revised: All hidden neurons and convolutional ﬁlters are rectiﬁed linear units (Nair & Hinton, 2010, ReLU ).

Original: We found that our approach achieves superior performance against strong baselines on all datasets.
Revised: We found that our approach achieves superior performance against strong baselines on all datasets.

Original: For all datasets, we hold out 20% of the training data as validation set for parameter tuning and model selection.
Revised: For all datasets, we hold out 20% of the training data as validation set for parameter tuning and model selection.

Original: After then, we combine both of these two sets to train the model and report the classiﬁcation error rate on test set.
Revised: After then, we combine both of these two sets to train the model and report the classiﬁcation error rate on test set.

Original: We optimize categorical cross-entropy loss on predicted class labels with
Revised: We optimize categorical cross-entropy loss on predicted class labels with

Original: Rademacher regularization.
Revised: Rademacher regularization.

Original: We update the parameters using mini-batch stochastic gradient descent with Nesterov momentum of 0 .
Revised: We update the parameters using mini-batch stochastic gradient descent with Nesterov momentum of 0 .

Original: 95 (Sutskever et al., 2013).
Revised: 95 (Sutskever et al., 2013).

Original: For Rademacher complexity term, we perform a grid search on the regularization weight λ ∈ { 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 .
Revised: For Rademacher complexity term, we perform a grid search on the regularization weight λ ∈ { 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 .

Original: 001 } , and update the dropout rates after every I ∈ { 1 , 5 , 10 , 50 , 100 } minibatches.
Revised: 001 } , and update the dropout rates after every I ∈ { 1 , 5 , 10 , 50 , 100 } minibatches.

Original: For variational dropout method (Kingma et al., 2015, V AR D ROP ), we examine the both Type-A and Type-B variational dropout with per-layer, per-neuron or per-weight adaptive dropout rate.
Revised: For variational dropout method (Kingma et al., 2015, V AR D ROP ), we examine the both Type-A and Type-B variational dropout with per-layer, per-neuron or per-weight adaptive dropout rate.

Original: We found the neuron-wise adaptive regularization on Type-A variational dropout layer often reports the best performance under most cases.
Revised: We found the neuron-wise adaptive regularization on Type-A variational dropout layer often reports the best performance under most cases.

Original: We also perform a grid search on the regularization noise parameter in { 0 . 1 , 0 . 01 , 0 .
Revised: We also perform a grid search on the regularization noise parameter in { 0 . 1 , 0 . 01 , 0 .

Original: 001 , 1 e − 4 , 1 e − 5 , 1 e − 6 } .
Revised: 001 , 1 e − 4 , 1 e − 5 , 1 e − 6 } .

Original: For sparse variational dropout method (Molchanov et al., 2017, S PARSE V AR D ROP ), we ﬁnd the model is much more sensitive to regularization weights, and often gets diverged.
Revised: For sparse variational dropout method (Molchanov et al., 2017, S PARSE V AR D ROP ), we ﬁnd the model is much more sensitive to regularization weights, and often gets diverged.

Original: We examine different regularization weight in { 1 e − 3 , 1 e − 4 , 1 e − 5 } .
Revised: We examine different regularization weight in { 1 e − 3 , 1 e − 4 , 1 e − 5 } .

Original: We follow similar weight adjustment scheme and scale it up by 10 after ﬁrst { 100 , 200 , 300 } epochs, then further scale up by 5 and 2 after same number of epoch.
Revised: We follow similar weight adjustment scheme and scale it up byafter ﬁrst { 100 , 200 , 300 } epochs, then further scale up by 5 and 2 after same number of epoch.

Original: Scales of Regularization In practice, we want to stablize regularization term within some managable variance, so its value does not vary signiﬁcantly upon difference structure of the underlying neural networks.
Revised: Scales of Regularization In practice, we want to stablize regularization term within some managable variance, so its value does not vary signiﬁcantly upon difference structure of the underlying neural networks.

Original: Hence, we design some heuristics to scale the regularizer to offset the multipler effects raised from network structure.
Revised: Hence, we design some heuristics to scale the regularizer to offset the multipler effects raised from network structure.

Original: For instance, recall the neural network deﬁned in Section 3, the
Revised: For instance, recall the neural network deﬁned in Section 3, the

Revised: To balance out the regularizer and the loss function so that they scale similarly as the sample size and internal layer nodes grow, we add some scaling factors to λ , which is discussed in Section 4.

Original: Rademacher complexity regularizer with p = q = 2 after scaling is
Revised: Rademacher complexity regularizer with p = q = 2 after scaling is

Original: [Equation]
Revised: [Figure]

Original: where W lj is the j th column of the weight coefﬁcient matrix W l and θ l is the retain rate vector for the l th layer.
Revised: where W lj is the j th column of the weight coefﬁcient matrix W l and θ l is the retain rate vector for the l th layer.

Original: The variable k is the number of classes to predict and X ∈ R n × d is the sample matrix.
Revised: The variable k is the number of classes to predict and X ∈ R n × d is the sample matrix.

Original: Similarly, we could rescale the Rademacher complexity regularizers under other settings of p = 1 , q = ∞ and p = ∞ , q = 1 .
Revised: Similarly, we could rescale the Rademacher complexity regularizers under other settings of p = 1 , q = ∞ and p = ∞ , q = 1 .

Original: Please refer to the appendix for the scaled Rademacher complexity bound regularizers and detailed derivations. 
Revised: Please refer to the appendix for the scaled Rademacher complexity bound regularizers and detailed derivations. [Figure]

Original: 4. MNIST
Revised: 4. MNIST

Original: MNIST dataset is a collection of 28 × 28 pixel hand-written digit images in grayscale, containing 60 K for training and 10 K for testing.
Revised: MNIST dataset is a collection of 28 × 28 pixel hand-written digit images in grayscale, containing 60 K for training and 10 K for testing.

Original: The task is to classify the images into 10 digit classes fromto 9 .
Revised: The task is to classify the images into 10 digit classes fromto 9 .

Original: All images are ﬂattened into 784 dimension vectors, and all pixel values are rescaled to gray scale.
Revised: All images are ﬂattened into 784 dimension vectors, and all pixel values are rescaled to gray scale.

Original: We examine several different network structures, including architectures of 1 hiddel layer with 1024 units, 2 hidden layers with 800 neurons each, as well as 3 hidden layers with 1024 units each.[Figure]
Revised: We examine several different network structures, including architectures of 1 hiddel layer with 1024 units, 2 hidden layers with 800 neurons each, as well as 3 hidden layers with 1024 units each.

Original: Table 1 compares the performance of our proposed models against other techniques.
Revised: Table 1 compares the performance of our proposed models against other techniques.

Original: We use a learning rate of 0 .and decay it by 0 .
Revised: We use a learning rate of 0 . 01 and decay it by 0 .

Original: 5 after every { 300 , 400 , 500 } epochs.
Revised: 5 after every { 300 , 400 , 500 } epochs.

Original: We let all models run sufﬁciently long with 100 K updates.
Revised: We let all models run sufﬁciently long with 100 K updates.

Original: For all models, we also explore different initialization for neuron retaining rates, including { 0 . 8 , 1 . 0 } for input layers, { 0 . 5 , 0 . 8 , 1 . 0 } for hidden layers.
Revised: For all models, we also explore different initialization for neuron retaining rates, including { 0 . 8 , 1 . 0 } for input layers, { 0 . 5 , 0 . 8 , 1 . 0 } for hidden layers.

Original: In practice, we ﬁnd initializing the retaining rates to 0 .
Revised: In practice, we ﬁnd initializing the retaining rates to 0 .

Original: 8 for input layer and 0 .
Revised: 8 for input layer and 0 .

Original: 5 for hidden layer yields better performance for all models, except for S PARSE V AR D ROP model, initializing retaining rate to 1 . 

Revised: 5 for hidden layer yields better performance for all models, except for S PARSE V AR D ROP model, initializing retaining rate to 1 .

Original: for input layer seems to give better result.
Revised: for input layer seems to give better result.

Original: Figure 2 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .
Revised: Figure 2 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .

Original: 01 regularization weight.
Revised: 01 regularization weight.

Original: The network contains one hidden layer of 1024 ReLU units.
Revised: The network contains one hidden layer of 1024 ReLU units.

Original: The retain rates were initialized to 0 .
Revised: The retain rates were initialized to 0 .

Original: 8 for input layer and 0 .
Revised: 8 for input layer and 0 .

Original: 5 for hidden layer.
Revised: 5 for hidden layer.

Original: The learning rate is 0 . 01 and decayed by 0 .
Revised: The learning rate is 0 . 01 and decayed by 0 .

Original: 5 after every 300 epochs.
Revised: 5 after every 300 epochs.

Original: We observe the retain rates for all layers are diffused throughout training process, and ﬁnally converged towards a unimodal distribution.
Revised: We observe the retain rates for all layers are diffused throughout training process, and ﬁnally converged towards a unimodal distribution.

Original: We also notice that the retain rates for input layer upon model convergence demonstrate interesting feature pattern of the dataset.
Revised: We also notice that the retain rates for input layer upon model convergence demonstrate interesting feature pattern of the dataset.

Original: For example, the pixels in surrounding margins yield smaller retain rates, and the center pixels often have larger retain rates.
Revised: For example, the pixels in surrounding margins yield smaller retain rates, and the center pixels often have larger retain rates.

Original: This is because the digits in MNIST dataset are often centered in the image, hence all the surrounding pixels are not predictive at all when classifying an instance.
Revised: This is because the digits in MNIST dataset are often centered in the image, hence all the surrounding pixels are not predictive at all when classifying an instance.

Original: This demonstrates that our proposed method is able to dynamically determine if an input signal is informational or not, and subsequently gives higher retain rate if it is, otherwise reduce the retain rate over time.
Revised: This demonstrates that our proposed method is able to dynamically determine if an input signal is informational or not, and subsequently gives higher retain rate if it is, otherwise reduce the retain rate over time.

Original: 4. CIFAR
Revised: 4. CIFAR

Original: CIFAR 10 and CIFAR 100 datasets are collections of 50 K training and 10 K testing RGB images from 10 and 100 different image categories. Every instance consists of 32 × 32 RGB pixels. We preprocess all images by subtracting the per-pixel mean computed over all training set, then with ZCA whitening as suggested in Srivastava et al. No data augmentation is used.
Revised: [Table]

Original: The neural network architecture we evaluate on uses three convolutional layers, each of which followed by a max-pooling layer. The convolutional layers have 96 , 128 , and 256 ﬁlters respectively. Each convolutional layer has a 5 × 5 receptive ﬁeld applied with a stride of 1 pixel, and each max-pooling layer pools from 3 ×
Revised: Convolutional

Original: Model CIFAR CIFAR Convolutional neural network 18 .50 .+
Revised: [Table]

Original:  Dropout in fully-connected 17 .45 .+
Revised: 18 .50 .+ Dropout in fully-connected 17 .45 .+

Original: V AR D ROP 16 .45 .+
Revised: V AR D ROP 16 .45 .+

Original: S PARSE V AR D ROP 17 .45 .+
Revised: S PARSE V AR D ROP 17 .45 .+

Original: Rademacher p = q =16 .44 .+
Revised: Rademacher p = q =16 .44 .+

Original: Rademacher p = 1 , q = inf 16 .45 .+
Revised: Rademacher p = 1 , q = inf 16 .45 .+

Original: Rademacher p = inf , q =16 .45 .+
Revised: Rademacher p = inf , q =16 .45 .+

Original: Dropout in all layers 15 .41 .+
Revised: Dropout in all layers 15 .41 .+

Original: V AR D ROP 15 .

Revised: V AR D ROP 15 .

Original: .+ S PARSE V AR D ROP 15 .42 .+
Revised: .+ S PARSE V AR D ROP 15 .42 .+

Original: Rademacher p = q =13 .38 .+
Revised: Rademacher p = q =13 .38 .+

Original: Rademacher p
Revised: Rademacher p

Original: =
Revised: =

Original: 1 , q = ∞ 13 .38 .+
Revised: 1 , q = ∞ 13 .38 .+

Original: Rademacher p = ∞ , q =13 .38 .
Revised: Rademacher p = ∞ , q =13 .38 .

Revised: CIFAR 10 and CIFAR 100 datasets are collections of 50 K training and 10 K testing RGB images from 10 anddifferent image categories.

Revised: Every instance consists of 32 × 32 RGB pixels.

Revised: We preprocess all images by subtracting the per-pixel mean computed over all training set, then with ZCA whitening as suggested in Srivastava et al.

Revised: No data augmentation is used.

Revised: The neural network architecture we evaluate on uses three convolutional layers, each of which followed by a max-pooling layer.

Original:  Table 2 summarizes the performance of our proposed models against other baselines.
Revised: The convolutional layers have 96 , 128 , and 256 ﬁlters respectively. Each convolutional layer has a 5 × 5 receptive ﬁeld applied with a stride of 1 pixel, and each maxpooling layer pools from 3 × 3 pixel region with strides of 2 pixels. These convolutional layers are followed by two fully-connected layer having 2048 hidden units each. Table 2 summarizes the performance of our proposed models against other baselines.

Original: We initialize dropout rates settings with { 0 . 9 , 1 . 0 } for input layers, { 0 .
Revised: We initialize dropout rates settings with { 0 . 9 , 1 . 0 } for input layers, { 0 .

Original: 75 , 1 . 0 } for convolutional layers and { 0 . 5 , 1 . 0 }
Revised: 75 , 1 . 0 } for convolutional layers and { 0 . 5 , 1 . 0 }

Original: for fully-connected layers.
Revised: for fully-connected layers.

Original: Similary to the MNIST evaluation, we ﬁnd setting the corresponding retaining probabilities for input layers, convolutional layers and fullyconnected layers to 0 .
Revised: Similary to the MNIST evaluation, we ﬁnd setting the corresponding retaining probabilities for input layers, convolutional layers and fully-connected layers to 0 .

Original: 75 and 0 .respectively yields best performance under all models.
Revised: 75 and 0 . 5 respectively yields best performance under all models.

Original: We initialize the learning rate to 0 .
Revised: We initialize the learning rate to 0 .

Original: 001 and decay it exponentially every { 200 , 300 , 400 } epochs.
Revised: 001 and decay it exponentially every { 200 , 300 , 400 } epochs.

Original: Figure 3 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .
Revised: Figure 3 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .

Original: 01 regularization weight.
Revised: 01 regularization weight.

Original: The network contains two convolution layers with 32 and 64 convolutional ﬁlters followed by one fully-connected layer with 1024 neurons.
Revised: The network contains two convolution layers with 32 and 64 convolutional ﬁlters followed by one fully-connected layer with 1024 neurons.

Original: All hidden units use ReLU activation functions.
Revised: All hidden units use ReLU activation functions.

Original: The retain rates were initialized to 0 .
Revised: The retain rates were initialized to 0 .

Original: 9 for input layer, 0 .
Revised: 9 for input layer, 0 .

Original: 75 for convolutional layer and 0 .
Revised: 75 for convolutional layer and 0 .

Original: 5 for fully-connected layer.
Revised: 5 for fully-connected layer.

Original: The learning rate is 0 .
Revised: The learning rate is 0 .

Original: 001 and exponentially decayed by half after every 300 epochs.
Revised: 001 and exponentially decayed by half after every 300 epochs.

Original: Similar to MNIST dataset, we observe the retain rates for all layers are diffused throughout training process, and ﬁnally converged towards a unimodal distribution.
Revised: Similar to MNIST dataset, we observe the retain rates for all layers are diffused throughout training process, and ﬁnally converged towards a unimodal distribution.

Original: However, unlike MNIST dataset, we do not see similar pattern for retain rates of input layer.
Revised: However, unlike MNIST dataset, we do not see similar pattern for retain rates of input layer.

Original: This is mainly due to the nature of dataset, such that CIFAR 10 images spread over the entire range, hence all pixels are potentially informational to the classiﬁcation process.
Revised: This is mainly due to the nature of dataset, such that CIFAR 10 images spread over the entire range, hence all pixels are potentially informational to the classiﬁcation process.

Original: This again demonstrates that the Rademacher regularizer is able to distinguish the informational pixels and retain them during training. 
Revised: This again demonstrates that the Rademacher regularizer is able to distinguish the informational pixels and retain them during training. [Figure]

Revised: of these channel pixels,

Revised: since they are all informational towards prediction.

Original: 4. T EXT C LASSIFICATION
Revised: 4. T EXT C LASSIFICATION

Original: In addition, we also compare our proposed approach on text classiﬁcation datasets—S UBJ and IMD B .
Revised: In addition, we also compare our proposed approach on text classiﬁcation datasets—S UBJ and IMD B .

Original: S UBJ is a dataset containing 10 K subjective and objective sentences (Pang & Lee, 2004) with nearly 14 .
Revised: S UBJ is a dataset containing 10 K subjective and objective sentences (Pang & Lee, 2004) with nearly 14 .

Original: 5 K vocabulary after stemming.
Revised: 5 K vocabulary after stemming.

Original: All subjective comments come from movie reviews expressing writer’s opinion, whereas objective sentences are from movie plots expressing purely facts.
Revised: All subjective comments come from movie reviews expressing writer’s opinion, whereas objective sentences are from movie plots expressing purely facts.

Original: We randomly sample 20% from the collections as test data, and use other 80% for training and validation.
Revised: We randomly sample 20% from the collections as test data, and use other 80% for training and validation.

Original: IMD B is a collection of movie reviews from IMD B website, with 25 K for training and another 25 K for test (Maas et al., 2011), containing more than 50 K vocabulary after stemming.
Revised: IMD B is a collection of movie reviews from IMD B website, with 25 K for training and another 25 K for test (Maas et al., 2011), containing more than 50 K vocabulary after stemming.

Original: It contains an even number of positive (i.e., with a review score of 7 or more out of a scale of 10 ) and negative (i.e., with a review score of 4 or less out of 10 ) reviews.
Revised: It contains an even number of positive (i.e., with a review score of 7 or more out of a scale of 10 ) and negative (i.e., with a review score of 4 or less out of 10 ) reviews.

Original: The dataset has a good movie diversity coverage with less than 30 reviews per movie.
Revised: The dataset has a good movie diversity coverage with less than 30 reviews per movie.

Original: For each sentence or document in these datasets, we normalize it into a vector of probability distribution over all vocabulary.
Revised: For each sentence or document in these datasets, we normalize it into a vector of probability distribution over all vocabulary.

Original: Table 3 summarizes the performance of our proposed models against other baselines.
Revised: Table 3 summarizes the performance of our proposed models against other baselines.

Original: We initialize dropout rates settings with { 0 . 8 , 1 . 0 } for input layers and { 0 . 5 , 1 . 0 } for fully-connected layers.
Revised: We initialize dropout rates settings with { 0 . 8 , 1 . 0 } for input layers and { 0 . 5 , 1 . 0 } for fully-connected layers.

Original: Similarly, by setting the corresponding retaining probabilities for input layers and fully-connected layers to 0 . 8 and 0 .
Revised: Similarly, by setting the corresponding retaining probabilities for input layers and fully-connected layers to 0 . 8 and 0 .

Original: 5 respectively, the model often yields the best performance.
Revised: 5 respectively, the model often yields the best performance.

Original: We use a constant learning rate of 0 .
Revised: We use a constant learning rate of 0 .

Original: 001 , as well as an initialization learning rate of 0 .
Revised: 001 , as well as an initialization learning rate of 0 .

Original: 01 and decay it by half every { 200 , 300 , 400 } epochs.
Revised: 01 and decay it by half every { 200 , 300 , 400 } epochs.

Original: We notice that overall the improvement of dropout is not as signiﬁcant as
Revised: We notice that overall the improvement of dropout is not as signiﬁcant as

Original: MNIST or CIFAR datasets.
Revised: MNIST or CIFAR datasets.

Original: Figure 4 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .
Revised: Figure 4 illustrates the changes in retain rates for both input and hidden layers under Rademacher regularization ( p = q = 2 ) with 0 .

Original: 005 regularization weight on IMD B dataset.
Revised: 005 regularization weight on IMD B dataset.

Original: The network contains one hidden layer of 1024 ReLU units.
Revised: The network contains one hidden layer of 1024 ReLU units.

Original: The retain rates were initialized to 0 .
Revised: The retain rates were initialized to 0 .

Original: 8 for input layer and 0 .for
Revised: 8 for input layer and 0 .for

Original: hidden layer.
Revised: hidden layer.

Original: The learning rate is 0 . 001 and decayed by 0 .
Revised: The learning rate is 0 . 001 and decayed by 0 .

Original: 5 after every 200 epochs.
Revised: 5 after every 200 epochs.

Original: Similar to
Revised: Similar to

Original: [Figure]

Original: there is no clear pattern from the retain rates out of these channel pixels, since they are all informational towards prediction.

Original: MNIST datasets, we observe the retain rates for all layers are diffused slightly, and the retain rates for input layer upon model convergence demonstrate interesting feature pattern of the dataset.[Table] + Dropout [Table]
Revised: MNIST datasets, we observe the retain rates for all layers are diffused slightly, and the retain rates for input layer upon model convergence demonstrate interesting feature pattern of the dataset.

Original: +

Original: [Equation]

Original: ROP

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: PARSE

Original: [Equation]

Original: AR

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] Recall that the task for IMD B dataset is to classify movie reviews into negative or positive labels.
Revised: Recall that the task for IMD B dataset is to classify movie reviews into negative or positive labels.

Original: Generically speaking, adjectives are more expressive than nouns or verbs in this scenario, and our ﬁndings seems to be consistent with this intuition.
Revised: Generically speaking, adjectives are more expressive than nouns or verbs in this scenario, and our ﬁndings seems to be consistent with this intuition.

Original: From our model, words like “lousi(y)” , “ﬂawless” , “obnoxi(ous)” , “ﬁnest” and “unwatch(able)” yield large retain rates and hence indicative feature to predication.
Revised: From our model, words like “lousi(y)” , “ﬂawless” , “obnoxi(ous)” , “ﬁnest” and “unwatch(able)” yield large retain rates and hence indicative feature to predication.

Original: We also notice that “baldwin” and “kurosawa” are also very informative features.
Revised: We also notice that “baldwin” and “kurosawa” are also very informative features.

Original: On the other hand, words like “young” , “review” , “role” , “anim(ation)” and “year” have near zero retain rates upon model convergence, which are less informative.
Revised: On the other hand, words like “young” , “review” , “role” , “anim(ation)” and “year” have near zero retain rates upon model convergence, which are less informative.

Original: One other interesting observation is that the word “oscar” also yields near zero retain rate, which implies the positivity or negativity of a movie review is not necessarily correlated with the mention of Academy Awards. 
Revised: One other interesting observation is that the word “oscar” also yields near zero retain rate, which implies the positivity or negativity of a movie review is not necessarily correlated with the mention of Academy Awards. [Figure] o f n e u r o n s [Figure]

Revised: alright make tree total complet finest zombi act made thing oscar call year unwatch kurosawaflawless forgett anim obnoxi actual mst3kunfunni unconvincbaldwin guy surprisingli lousi role refresh

Revised: [Figure]

Original:   C ONCLUSION
Revised: some genre and generic movie plot information “act” , “anim(ation)” , “guy” , “kill” , “music” , “role” , and “zombi” . One interesting observation is that we ﬁnd the word “oscar” is also in the list of less informative features, which implies movie reviews and Academy Awards are not necessarily correlated. Note that higher retaining rate means the corresponding features are more indicative in classifying IMD B reviews into positive and negative labels, i.e., no explicit association with the label itself.  C ONCLUSION

Original: Imposing regularizaiton for a better model generalization is not a new topic.
Revised: Imposing regularizaiton for a better model generalization is not a new topic.

Original: However we tackle the problem for the dropout neural network regularization in a different way.
Revised: However we tackle the problem for the dropout neural network regularization in a different way.

Original: The theoretical upper bound we proved on the Rademacher complexity facilitates us to directly incorporate the dropout rates into the objective function.
Revised: The theoretical upper bound we proved on the Rademacher complexity facilitates us to directly incorporate the dropout rates into the objective function.

Original: In this way the dropout rate can be optimized by block coordinate [Figure]
Revised: In this way the dropout rate can be optimized by block coordinate descent procedure with one consistent objective.

Original: (ated)” , “unfunni(y)” , and “unwatch(able)” .

Original: In addition, some actor, directory or show names also appear in the top informative word list, such as “baldwin” , “kurosawa” and “mst3k” .

Original: Some of the word features with low retaining probability—hence, possibly less indicative—include “actual” , “call” , “complete” , “make” , “pretti(y)” , “review” , “thing” , “total” , “year” , “young” .

Original: Moreover, some genre and generic movie plot information “act” , “anim(ation)” , “guy” , “kill” , “music” , “role” , and “zombi” .

Original: One interesting observation is that we ﬁnd the word “oscar” is also in the list of less informative features, which implies movie reviews and Academy Awards are not necessarily correlated.

Original: Note that higher retaining rate means the corresponding features are more indicative in classifying IMD B reviews into positive and negative labels, i.e., no explicit association with the label itself.

Original: descent procedure with one consistent objective.

Original: Our empirical evaluation demonstrates promising results and interesting patterns on adapted retain rates.
Revised: Our empirical evaluation demonstrates promising results and interesting patterns on adapted retain rates.

Original: In the future, we would like to investigate the sparsity property of the learnt retain rates to encourage a sparse representation of the data and the neural network structure (Wen et al., 2016), similar to the sparse Bayesian models and relevance vector machine (Tipping, 2001).
Revised: In the future we would like to investigate the sparsity property of the learnt retain rates to encourage a sparse representation of the data and the neural network structure (Wen et al., 2016), similar to the sparse Bayesian models and relevance vector machine (Tipping, 2001).

Original: We would also like to explore the applications of deep network compression (Han et al., 2015a;
Revised: We would also like to explore the applications of deep network compression (Han et al., 2015a;

Original: Iandola et al., 2016; Ullrich et al., 2017; Molchanov et al., 2017; Louizos et al., 2017).
Revised: Iandola et al., 2016; Ullrich et al., 2017; Molchanov et al., 2017; Louizos et al., 2017).

Original: In addition, one other possible research direction is to dynamically adjust the architecture of the deep neural networks (Srinivas & Babu, 2015; Han et al., 2015b; Guo et al., 2016), and hence reduce the model complexity via dropout rates.
Revised: In addition, one other possible research direction is to dynamically adjust the architecture of the deep neural networks (Srinivas & Babu, 2015; Han et al., 2015b; Guo et al., 2016) and hence reduce the model complexity via dropout rates

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Available after blind review.
Revised: Available after blind review.

Original:  A PPENDIX 6. P ROOF OF T HEOREM 3. 
Revised:  A PPENDIX 6. P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: In the analysis of Rademacher complexity, we treat the functions fed into the neurons of the l th layer as one function class F l = f l ( x ; w : l ) .
Revised: In the analysis of Rademacher complexity we treat the functions fed into the neurons of the l th layer as one function class F l = f l ( x ; w : l ) .

Original: Here again we are using the notation w : l = { w 1 , . . .
Revised: Here again we are using the notation w : l = { w 1 , . . .

Original: , w l } and w = w : L .
Revised: , w l } and w = w : L .

Original: As a consequence ∀ j, f lj ( x ; w l ) ∈ F l .
Revised: As a consequence ∀ j, f lj ( x ; w l ) ∈ F l .

Revised: Note here f L ( x ; W ) used in section 3 is a vector, but f L ( x ; w ) used in this subsection is a scalar.

Revised: The connection between f L ( x ; W ) and f L ( x

Original: To simplify our analysis, we follow Wan et al.
Revised: To simplify our analysis, we follow Wan et al.

Original: and reformulate the cross-entropy loss on top of the softmax into a single logistic function
Revised: and reformulate the cross-entropy loss on top of the softmax into a single logistic function

Original: [Equation]
Revised: [Equation]

Original: The function class fed into the neurons of the l th layer f l ( x ; w : l ) admits a recursive expression
Revised: The function class fed into the neurons of the l th layer f l ( x ; w : l ) admits a recursive expression

Original: [Equation]
Revised: [Equation]

Original: Given the neural network function (1) and the logistic loss function l is 1 Lipschitz, by Contraction lemma  (Shalev-Shwartz & Ben-David, 2014), the empirical Rademacher complexity of the loss function is bounded by
Revised: Given the neural network function (1) and the logistic loss function l is 1 Lipschitz, by Contraction lemma (a variant of the lemma 26.9 on page 381, Chapter 26 of (Shalev-Shwartz & Ben-David, 2014)), the empirical Rademacher complexity of the loss function is bounded by

Original: [Equation]
Revised: [Equation]

Original: Note the empirical Rademacher complexity of the function class of the L th layer, i.e., the last output layer, is
Revised: Note the empirical Rademacher complexity of the function class of the L th layer, i.e., the last output layer, is

Original: [Equation]
Revised: [Equation]

Original: To prove the bound in a recursive way, let’s also deﬁne a variant of the Rademacher complexity with absolute value inside the supremum:
Revised: To prove the bound in a recursive way, let’s also deﬁne a variant of the Rademacher complexity with absolute value inside the supremum:

Original: [Equation]
Revised: [Equation]

Original:  Note here ˜ R S ( f ) is not exactly the same as the Rademacher complexity deﬁned before in this paper.
Revised: [Equation] Note here ˜ R S ( f ) is not exactly the same as the Rademacher complexity deﬁned before in this paper.

Original: And we have
Revised: And we have

Revised: R S

Original: [Equation]
Revised: [Equation]

Revised: f L

Revised: [Equation]

Revised: [Equation]

Original: Now we start the recursive proof.
Revised: Now we start the recursive proof.

Original: The empirical Rademacher complexity (with absolute value inside supremum) of the function class of the l th layer is
Revised: The empirical Rademacher complexity (with absolute value inside supremum) of the function class of the l th layer is

Original: [Equation]
Revised: [Equation]

Original: Let g l − 1 j ( x ; w : l − 1 , θ :( l − 2) )
Revised: Let g l − 1 j ( x ; w : l − 1 , θ :( l − 2) )

Original: =
Revised: =

Original: E r :( l − 2) φ (cid:0) f l − 1 j ( x i ; w : l − 1 , r : l − 2 ) (cid:1) , then[Equation] [Equation]
Revised: E r :( l − 2) φ (cid:0) f l − 1 j ( x i ; w : l − 1 , r : l − 2 ) (cid:1) , then

Original: According to the assumption, p ≥

Original: [Equation]
Revised: [Equation]

Original: q = p/ ( p − 1)

Original: [Equation] and (cid:107) w l (cid:107) p ≤ B l , from equation (10) [Equation] by Holder’s inequality, we have ∀ l ∈ { 2 , . . .
Revised: According to the assumption, p ≥ 1 , q = p/ ( p − 1) , and (cid:107) w l (cid:107) p ≤ B l , from equation (10), by Holder’s inequality, we have ∀ l ∈ { 2 , . . .

Original: , L }
Revised: , L }

Original: [Equation]
Revised: [Equation]

Original: Suppose the activation function φ ( · ) used in the neural network is 1 -Lipschitz [Equation] and φ (0)
Revised: Suppose the activation function φ ( · ) used in the neural network is 1 -Lipschitz, and φ (0)

Original: = 

Revised: =

Original: (for example, the
Revised: (for example, the

Original: RELU function).
Revised: RELU function).

Original: Then by the Ledoux-Talagrand contraction lemma, ∀ l ∈ { 1 , . . .
Revised: Then by the Ledoux-Talagrand contraction lemma, ∀ l ∈ { 1 , . . .

Original: , L − 1 }˜
Revised: , L − 1 }

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the ﬁrst layer
Revised: For the ﬁrst layer

Original: , i
Revised: , i

Original: [Equation]
Revised: [Table]

Original: e
Revised: e., the feature layer with dropout but without activation function, if | w 1 | p ≤ B 1 , then

Original: [Equation]
Revised: [Equation]

Original: , the feature layer with dropout but without activation function, if | w 1 | p ≤ B 1 , then
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: By Lemma 26.11 in Shalev-Shwartz & Ben-David (2014), we know the last term in (13) is bounded by
Revised: By Lemma 26.11 on page 383, Chapter 26.2 of Shalev-Shwartz & Ben-David (2014), we know the last term in (13) is bounded by

Original: [Equation]
Revised: [Equation]

Original: Thus we get
Revised: Thus we get

Original: [Equation]
Revised: [Equation]

Original: Combining the inequalities (5), (8), (11), (12), and (14), we have
Revised: Combining the inequalities (5), (8), (11), (12), and (14), we have

Original: [Equation]
Revised: [Equation]

Original: 6. G ENERALIZATION B OUND ON THE D ROPOUT N EURAL N ETWORKS
Revised: 6. G ENERALIZATION B OUND ON THE D ROPOUT N EURAL N ETWORKS

Original: Here we need to deﬁne truncated cross entropy loss function:
Revised: Here we need to deﬁne truncated cross entropy loss function:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where C l is a constant.
Revised: (16) where C l is a constant.

Original: Note with the truncation, the cross entropy loss is still 1 -Lipschitz
Revised: Note with the truncation, the cross entropy loss is still 1 -Lipschitz

Original: so the empirical
Revised: so the empirical

Original: Rademacher complexity bound still holds for the truncated loss ˜ l ( f L ( x ; W , θ ) , y ) .
Revised: Rademacher complexity bound still holds for the truncated loss ˜ l ( f L ( x ; W , θ ) , y ) .

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: For the dropout neural network deﬁned in section (3), if truncated cross entropy loss ˜ l (16) is used, then ∀ δ ≥ 0 , with probability at least 1 − 2 δ , ∀ ˜ l ◦ f L :
Revised: For the dropout neural network deﬁned in section (3), if truncated cross entropy loss ˜ l (16) is used, then ∀ δ ≥ 0 , with probability at least 1 − 2 δ , ∀ ˜ l ◦ f L :

Original: [Equation]
Revised: [Equation]

Original: Note here the empirical Rademacher complexity for the bounded loss function R S (˜ l ◦ f L ) admits the same bound as the empirical Rademacher complexity for the unbounded cross entropy loss R S ( l ◦ f L ) .
Revised: Note here the empirical Rademacher complexity for the bounded loss function R S (˜ l ◦ f L ) admits the same bound as the empirical Rademacher complexity for the unbounded cross entropy loss R S ( l ◦ f L ) .

Original: 6. T OWARDS AN U NIFIED V IEW OVER R ADEMACHER R EGULARIZATION (cid:63)
Revised: 6. T OWARDS AN U NIFIED V IEW OVER R ADEMACHER R EGULARIZATION (cid:63)

Original: In fact, adding a Rademacher related regularizer, though not investigated much, is not new at least for linear functions.
Revised: In fact, adding a Rademacher related regularizer, though not investigated much, is not new at least for linear functions.

Original: It is well known (Shalev-Shwartz & Ben-David, 2014) that the empirical Rademacher complexity of the linear class
Revised: It is well known (Shalev-Shwartz & Ben-David, 2014) that the empirical Rademacher complexity of the linear class

Original: [Equation]
Revised: [Equation]

Original: is bounded by
Revised: is bounded by

Original: [Equation]
Revised: [Equation]

Original: Note the l 2 loss function is 2 -Lipschtz.
Revised: Note the l 2 loss function is 2 -Lipschtz.

Original: In this way, we may interpret the regularizer in the ridge regression related an upper bound for the empirical Rademacher complexity of the linear function class.
Revised: In this way, we may interpret the regularizer in the ridge regression related an upper bound for the empirical Rademacher complexity of the linear function class.

Original: Similarly for the linear class
Revised: Similarly for the linear class

Original: [Equation]
Revised: [Equation]

Original: the empirical Rademacher complexity is bounded by
Revised: the empirical Rademacher complexity is bounded by

Original: [Equation]
Revised: [Equation]

Original: So the lasso problem can also be viewed as adding a Rademacher-related regularization to the empirical loss minimization objective.
Revised: So the lasso problem can also be viewed as adding a Rademacher-related regularization to the empirical loss minimization objective.

Original: (cid:63) Note that the content in Section 6.3 and 6.4 is based purely on heuristics, and derived on an ad hoc basis.

Original: 6. S CALES OF R EGULARIZATION (cid:63)
Revised: 6. S CALES OF R EGULARIZATION (cid:63)

Original: In application we do not want the regularization term to vary too much when the neural network has different number of internal neurons.
Revised: In application we do not want the regularization term to vary too much when the neural network has different number of internal neurons.

Original: To overcome that we design some heuristics to add to the regularizer.
Revised: To overcome that we design some heuristics to add to the regularizer.

Original: Note here all the scales mentioned in this section are added in a heuristic fashion.
Revised: Note here all the scales mentioned in this section are added in a heuristic fashion.

Original: It is purely empirical.
Revised: It is purely empirical.

Original: When p = q = 2 , the regularizer is bounded by
Revised: When p = q = 2 , the regularizer is bounded by

Original: [Equation]
Revised: [Equation]

Original: where W lj ∈ R k l − 1 is the j -th column of W l .
Revised: where W lj ∈ R k l − 1 is the j -th column of W l .

Original: Suppose we use i.i.d.
Revised: Suppose we use i.i.d.

Original: uniform random variable to initialize W l such that ∀ l w lij ∼ U [ − (cid:112) 6 / ( k l + k l +1 ) , (cid:112) 6 / ( k l + k l +1 )] .
Revised: uniform random variable to initialize W l such that ∀ l w lij ∼ U [ − (cid:112) 6 / ( k l + k l +1 ) , (cid:112) 6 / ( k l + k l +1 )] .

Original: Considering the scales of the maximum among the 2 -norms of W lj , we use a scaled regularizer instead:
Revised: Considering the scales of the maximum among the 2 -norms of W lj , we use a scaled regularizer instead:

Original: [Equation]
Revised: [Equation]

Original: Similarly, when p = ∞ and q = 1 , the scaled regularizer we used is
Revised: Similarly, when p = ∞ and q = 1 , the scaled regularizer we used is

Original: [Equation]
Revised: [Equation]

Original: When p = 1 and q = ∞ , we used the scaled regularizer
Revised: When p = 1 and q = ∞ [Equation] we used the scaled regularizer

Original: [Equation]
Revised: [Equation]

Revised: 6. S TOCHASTIC O BJECTIVE AND ITS D ETERMINISTIC A PPROXIMATION

Revised: To optimize the model parameters and the retain rates, we use a block coordinate descent algorithm.

Revised: We random initialize W and θ , and optimize them in an alternating fashion.

Revised: For the retain rate probability θ , due to the stochastic nature of dropout framework, it is expensive to compute the estimation of the objective.

Revised: In our case, we use the expected value of the Bernoulli dropout variables and rescale the output from each layer, to approximate the true objective f L ( x ; W , θ ) .

Revised: It signiﬁcantly speeds up the forward propagation process, as we do not need to iteratively sample the dropout variables during optimization.

Revised: Essentially, it makes the layer output deterministic and the underlying network operates as if without dropout, which is exactly the (cid:63) Note

Revised: Section 6.

Revised: [Figure]

Revised: same as the approximation used in (Srivastava et al., 2014) during testing time.

Revised: Note that using the expected value of Bernoulli dropout random variables to rescale a layer output is an approximation to the true objective θ )

Revised: Figure 5 shows the changes of the true objective in “stochastic” mode and its “deterministic” approximation on the training set during optimization under different network settings.

Revised: Empirically, we observe that the true optimization objective in stochastic mode as in Eqn.

Revised: (2) decreases consistently if we use the expected value of the Bernoulli dropout random variable to approximate the sampling process.

Revised: 6. S TABILITY OF THE D ROPOUT R ATES C ONVERGENCE

Revised: In this sub-section we demonstrate the dropout rates convergence acrross multiple runs.

Revised: We use a neural network with one hidden layer of 1024 ReLU units to illustrate, and feed it with MNIST dataset.

Revised: We train the network for 10 different runs, with same conﬁgurations and empirical settings, except different initializations on the network weight coefﬁcients.

Revised: Figure 6 shows the histogram of hidden layer retain rates upon model convergence under different runs.

Revised: We observe similar dropout behavior and distribution among multiple runs upon model convergence, i.e., the histograms of the retain rates do not diverge much across different runs in regards to different model weight initializations.

Revised: [Figure]