Targeted Dropout
Abstract
Neural networks are extremely ﬂexible models due to their large number of parameters, which is beneﬁcial for learning, but also highly redundant.
This makes it possible to compress neural networks without having a drastic effect on performance.
We introduce targeted dropout, a strategy for post hoc pruning of neural network weights and units that builds the pruning mechanism directly into learning.
At each weight update, targeted dropout selects a candidate set for pruning using a simple selection criterion, and then stochastically prunes the network via dropout applied to this set.
The resulting network learns to be explicitly robust to pruning, comparing favourably to more complicated regularization schemes while at the same time being extremely simple to implement, and easy to tune.
Introduction
There has been a great deal of work on developing strategies to sparsify neural networks [10, 7, 5, 2, 13, 3, 12, 9, 18].
Sparsiﬁcation involves removing weights (corresponding to setting them to 0 ) or entire units from the network, while maintaining predictive performance.
Sparsity can be encouraged during learning by the use of sparsity-inducing regularizers, like L 1 or L 0 penalties.
It can also be imposed by post hoc pruning, where a full-sized network is trained and then sparsiﬁed according to some pruning strategy.
Ideally, given some measurement of task performance, we would prune the weights or units that provide the least amount of beneﬁt to the task.
Finding the optimal set is in general a difﬁcult combinatorial problem and even a greedy strategy would require an unrealistic number of task evaluations, as there are often millions of parameters in a modern neural network architecture.
Common pruning strategies therefore focus on fast approximations, such as removing weights with the smallest magnitude [6], or ranking the weights by the sensitivity of the task performance with respect to the weights and removing the least-sensitive ones [10].
Our approach is based on the observation that dropout regularization [8, 17] itself enforces sparsity during training, by sparsifying the network with each forward pass.
This encourages the network to learn a representation that is robust to a particular form of post hoc sparsiﬁcation – in this case, where a random set of units are removed.
Our hypothesis is that if we plan to do explicit post hoc sparsiﬁcation, then we can do better by speciﬁcally applying dropout to the set of units that we a priori believe are the least useful.
We call this approach targeted dropout .
The idea is to rank weights or units according to some fast, approximate measure of importance (like magnitude), and then apply dropout primarily to those elements with low importance .
Similar to the observation with regular dropout, we show that this encourages the network to learn a representation where the importance of weights or units more closely aligns with our approximation.
In other words, the network learns to be robust to our choice of post hoc pruning strategy.
The advantage of targeted dropout compared to other approaches is that it leads to a converged network that is extremely robust to post hoc pruning.
It is simultaneously easy to implement, consisting of a two-line change using neural network frameworks such as Tensorﬂow [1] or PyTorch [14].
Furthermore, it is explicit: the desired level of sparsity is provided by the user, and enforced throughout training.
Targeted Dropout 2.1 Dropout
Our work uses the two most popular Bernoulli dropout techniques, Hinton et al.
’s unit dropout [8, 17] and Wan et al.
’s
weight dropout (dropconnect) [20].
For a fully-connected layer with input tensor
XXX , weight matrix W , output tensor YYY , and mask M i,o ∼ Bernoulli( α ) we deﬁne both techniques below:
Unit dropout [8, 17]:
[Equation]
Unit dropout randomly drops units (sometimes referred to as neurons) at each training step to reduce dependence between units and prevent overﬁtting.
Weight dropout [20]:
[Equation]
Weight dropout randomly drops individual weights in the weight matrices at each training step.
Intuitively, this is dropping connections between layers, forcing the network to adapt to a different connectivity at each training step.
2.2 Magnitude-based pruning
A popular class of pruning strategies are those characterized as magnitude-based pruning strategies.
These strategies treat the top- k largest magnitude weights as important.
We use argmax-k to return the top- k elements (units or weights) out of all elements being considered.
Unit pruning [6]: considers the units (column-vectors) of weight matrices under the L 2 -norm.
[Equation]
Weight pruning [10]: considers each entry of the
[Equation]
While weight pruning tends to preserve more of the task performance under coarser prunings [5, 19, 4], unit pruning allows for considerably greater computational savings [21, 12].
2.3 Methods
Consider a neural network parameterized by θθθ , and our pruning strategy (deﬁned above in Equations (1) and (2))
W .
We hope to ﬁnd optimal parameters θθθ ∗ such that our loss E ( W ( θθθ ∗ )) is low and at the same time |W ( θθθ ∗ ) | ≤ k , i.e. we wish to keep only the k weights of highest magnitude in the network.
A deterministic implementation would select the bottom | θθθ | − k elements and drop them out.
However, we would like for low-valued elements to be able to increase their value if they become important during training.
Therefore, we introduce stochasticity into the process using a targeting proportion γ and a drop probability α .
The targeting proportion means that we select the bottom γ | θθθ | weights as candidates for dropout, and of those we drop the elements independently with drop rate α .
This implies that the expected number of units to keep during each round of targeted dropout is (1 − γ · α ) | θ θ θ | .
As we will see below, the result is a reduction in the important subnetwork dependency on the unimportant subnetwork, thereby reducing the performance degradation of pruning at the conclusion of training.
Our weight pruning experiments (shown in Tables 1 & 2) demonstrate that the baseline regularization schemes are comparatively weak to their targeted counterparts.
We ﬁnd that targeted dropout applied
[Table]
Dropout
[Table] [Table]
to the weights results in the network outperforming unregularized performance with only half the total number of parameters.
We are able to reach extremely high rates of pruning in Table 2 by gradually annealing the targeted proportion from zero percent of weights up to ninety-nine percent throughout the course of training.
Conclusion
We propose targeted dropout as a simple and extremely effective regularization tool for incorporating post hoc pruning strategies into the training procedure of neural networks without drastic impact to underlying task performance of a particular architecture.
Among the primary beneﬁts of targeted dropout are the simplicity of implementation and intuitive, ﬂexible hyperparameters. Appendix Dependence Between the Important and Unimportant Subnetworks
The goal of targeted dropout is to reduce the dependence of the important subnetwork on its complement.
A commonly used intuition behind dropout is the prevention of coadaptation between units; that is, when dropout is applied to a unit, the remaining network can no longer depend on that unit’s contribution to the function and must learn to propagate that unit’s information through a more reliable channel.
An alternative description asserts that dropout maximizes the mutual information between units in the same layer, thereby decreasing the impact of losing a unit [17].
Dropout has also been shown to impose ‘hierarchy’ among units depending on the particular drop rate associated with each unit [15].
A more relevant intuition into the effect of targeted dropout in our speciﬁc pruning scenario can be obtained from an illustrative case where the important subnetwork is completely separated from the unimportant one.
Suppose a network was composed of two non-overlapping subnetworks, each able to produce the correct output by itself, with the network output given as the average of both subnetwork outputs.
If our importance criterion designated the ﬁrst subnetwork as important, and the second subnetwork as unimportant (more speciﬁcally, it has lower weight magnitude), then adding noise to the weights of the unimportant subnetwork (i.e. applying dropout) means that with non-zero probability we will corrupt the network output.
Since the important subnetwork is already able to predict the output correctly, to reduce the loss we must therefore reduce the weight magnitude of the unimportant subnetwork output layer towards zero, in effect “killing” that subnetwork, and reinforcing the separation between the important subnetwork and the unimportant one.
These interpretations make clear why dropout should be considered a natural tool for application in pruning.
We can empirically conﬁrm targeted dropout’s effect on weight dependence by comparing a network trained with and without targeted dropout and inspecting the Hessian and gradient to determine the dependence of the network on the losing ticket (i.e. our weights to be pruned).
As in
LeCun et al.
[10], we can estimate the effect of pruning weights by considering the second degree
Taylor expansion of change in loss:
[Equation]
Where d i
=
θ i if θ i ∈ W ( θθθ ) and 0 otherwise.
∇ θθθ E are the gradients of the loss, and HHH is the Hessian.
Note that at the end of training, if we have found a critical point θθθ ∗ , then ∇ θθθ E ( θθθ ∗ )
= 000 , leaving only the Hessian term.
In our experiments we empirically conﬁrm that targeted dropout reduces the dependence between the important and unimportant subnetworks by an order of magnitude (See Fig.
Weight Dropout/Pruning
[Table]
LeNet-
Our ﬁrst suite of experiments are performed on the LeNet-5 architecture [11] applied to the CIFARdataset.
This architecture is used as a control as it is comparatively shallow and underperforming relative to the ResNet architecture used in the next section.
The network is trained for 256 epochs by
SGD with momentum of 0.9 and a constant learning rate of 0.01.
We apply basic input augmentation in the form of random crops, random horizontal ﬂipping, and standardization.
In Table 3 we compare both weight and unit targeted dropout against standard weight and unit dropout, as well as L 1 regularization, variational dropout, and L 0 regularization.
We ﬁnd that both forms of targeted dropout outperform their standard dropout counterpart; we also ﬁnd that the version of targeted dropout used (unit versus weight) must match the version of pruning strategy used post hoc.
Variational dropout performs fairly well in both its weight and unit forms, however the sparsity is effectively ﬁxed at the rate discovered by the optimization procedure, and the model accuracy breaks down rapidly once this threshold is surpassed.
The weight-level variant of L 0 regularization performs very well on LeNet and is capable of sustaining signiﬁcant pruning; however, targeted dropout is superior for all rates over 50% aside from 90%.
The unit-level variant performs only marginally better than the unregularized baseline, and is signiﬁcantly outperformed by targeted dropout.
Regarding Lregularization, we ﬁnd that it doesn’t seem to improve over the unregularized baseline.
VGG-
In order to experiment with a different architecture/data pair we experiment with the VGGarchitecture [16] on CIFAR-100.
We ﬁnd that targeted dropout behaves similarly across all datasetarchitecture pairs (see Table 4).
[Table]