Pay Attention to Features, Transfer Learn Faster CNNs
Abstract
Deep convolutional neural networks are now widely deployed in vision applications, but a limited size of training data can restrict their task performance.
Transfer learning oﬀers the chance for CNNs to learn with limited data samples by transferring knowledge from models pretrained on large datasets.
Blindly transferring all learned features from the source dataset, however, brings unnecessary computation to CNNs on the target task.
In this paper, we propose attentive feature distillation and selection (AFDS), which not only adjusts the strength of transfer learning regularization but also dynamically determines the important features to transfer.
By deploying AFDS on ResNet-101, we achieved a state-of-the-art computation reduction at the same accuracy budget, outperforming all existing transfer learning methods.
With a 10 × MACs reduction budget, a ResNetequipped with AFDS transfer learned from ImageNet to Stanford Dogs 120, can achieve an accuracy 11 .
07% higher than its best competitor.
Introduction
Despite recent successes of CNNs achieving state-of-the-art performance in vision applications (Tan & Le, 2019; Cai & Vasconcelos, 2018; Zhao et al., 2018; Ren et al., 2015), there are two major shortcomings limiting their deployments in real life.
First, training CNNs from random initializations to achieve high task accuracy generally requires a large amount of data that is expensive to collect.
Second, CNNs are typically compute-intensive and memory-demanding, hindering their adoption to power-limited scenarios.
To address the former challenge, transfer learning (Pan & Yang, 2009) is thus designed to transfer knowledge learned from the source task to a target dataset that has limited data samples.
In practice, we often choose a source dataset such that the input domain of the source comprises the domain of the target.
A common paradigm for transfer learning is to train a model on a large source dataset, and then ﬁne-tune the pre-trained weights with regularization methods on the target dataset (Zagoruyko & Komodakis, 2017;
Yim et al., 2017; Li et al., 2018; Li & Hoiem, 2018; Li et al., 2019).
For example, one regularization method, L 2 -SP (Li et al., 2018), penalizes the L 2 -distances of pretrained weights on the source dataset and the weights being trained on the target dataset.
The pretrained source weights serves as a starting point when training on the target data.
During ﬁne-tuning on the target dataset, the regularization constrains the search space around this starting point, which in turn prevents overﬁtting the target dataset.
Intuitively, the responsibility of transfer learning is to preserve the source knowledge acquired by important neurons.
The neurons thereby retain their abilities to extract features from the source domain, and contribute to the network’s performance on the target dataset.
Moreover, by determining the importance of neurons, unimportant ones can further be removed from computation during inference with network pruning methods (Luo et al., 2017;
He et al., 2017; Zhuang et al., 2018; Ye et al., 2018; Gao et al., 2019).
The removal of unnecessary compute not only makes CNNs smaller in size but also reduces computational costs while minimizing possible accuracy degradations.
As the source domain encompasses the target, many neurons responsible for extracting features from the source domain may become irrelevant to the target domain and can be removed.
In Figure 1, a simple empirical study of the channel neurons’ activation magnitudes corroborates our intuition: as deeper layers extract higher-level features, more neurons become either specialized or irrelevant to dogs.
The discussion above hence prompts two questions regarding the neurons: which neurons should we transfer source knowledge to, and which are actually important to the target model?
[Figure]
Yet traditional transfer learning methods fail to provide answers to both, as generally they transfer knowledge either equally for each neuron with the same regularized weights, or determine the strength of regularization using only the source dataset (Li et al., 2018).
The source domain could be vastly larger than the target, giving importance to weights that are irrelevant to the target task.
Recent years have seen a surge of interest in network pruning techniques, many of which induce sparsity by pushing neuron weights or outputs to zeros, allowing them to be pruned without a detrimental impact on the task accuracies.
Even though pruning methods present a solution to neuron/weight importance, unfortunately they do not provide an answer to the latter question, i.e. whether these neurons/weights are important to the target dataset.
The reason for this is that pruning optimization objectives are often in conﬂict with traditional transfer learning, as both drive weight values in diﬀerent directions: zero for pruning and the initial starting point for transfer learning.
As we will see later, a na¨ıve composition of the two methods could have a disastrous impact on the accuracy of a pruned CNN transferlearned on the target dataset.
In this paper, to tackle the challenge of jointly transferring source knowledge and pruning target CNNs, we propose a new method based on attention mechanism (Vaswani et al., 2017), attentive feature distillation and selection (AFDS).
For the images in the target dataset, AFDS dynamically learns not only the features to transfer, but also the unimportant neurons to skip.
During transfer learning, instead of ﬁne-tuning with L 2 -SP regularization which explores the proximity of the pre-trained weights, we argue that a better alternative is to mimic the feature maps, i.e. the output response of each convolutional layer in the source model when images from the target dataset are shown, with L 2 -distances.
This way the ﬁne-tuned model can still learn the behavior of the source model.
Additionally, without the restriction of searching only the proximity of the initial position, the weights in the target model can be optimized freely and thus increasing their generalization capacity.
Therefore, we present attentive feature distillation (AFD) to learn which relevant features to transfer.
To accelerate the transfer-learned model, we further propose attentive feature selection (AFS) to prune networks dynamically.
AFS is designed to learn to predictively select important output channels in the convolution to evaluate and skip unimportant ones, depending on the input to the convolution.
Rarely activated channel neurons can further be removed from the network, reducing the model’s memory footprint.
From an informal perspective, both AFD and AFS learn to adjust the “valves” that control the ﬂow of information for each channel neuron.
The former adjusts the strength of regularization, thereby tuning the ﬂow of knowledge being transferred from the source model.
The latter allows salient information to pass on to the subsequent layer and stops the ﬂow of unimportant information.
A signiﬁcant attribute that diﬀerentiates AFD and AFS from their existing counterparts is that we employ attention mechanisms to adaptively learn to “turn the valves” dynamically with small trainable auxiliary networks.
Our main contributions are as follows:
• We present attentive feature distillation and selection (AFDS) to eﬀectively transfer learn CNNs, and demonstrate state-of-the-art performance on many publicly available datasets with ResNet-101 (He et al., 2016) models transfer learned from ImageNet (Deng et al., 2009).
• We paired a large range of existing transfer learning and network pruning methods, and examined their abilities to trade-oﬀ FLOPs with task accuracy.
• By changing the fraction of channel neurons to skip for each convolution, AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy.
We found that AFDS generally provides the best FLOPs and accuracy trade-oﬀ when compared to a broad range of paired methods.
Related Work 2. Transfer Learning
Training a deep CNN to achieve high accuracy generally require a large amount of training data, which may be expensive to collect.
Transfer learning (Pan & Yang, 2009) addresses this challenge by transferring knowledge learned on a large dataset that has a similar domain to the training dataset.
A typical approach for CNNs is to ﬁrst train the model on a large source dataset, and make use of their feature extraction abilities (Donahue et al., 2014;
Razavian et al., 2014).
Moreover, it has been demonstrated that the task accuracy can be further improved by ﬁne-tuning the resulting pre-trained model on a smaller target dataset with a similar domain but a diﬀerent task (Yosinski et al., 2014; Azizpour et al., 2015).
Li et al.
(2018) proposed L 2 -SP regularization to minimize the L 2 -distance between each ﬁne-tuned parameter and its initial pre-trained value, thus preserving knowledge learned in the pre-trained model.
In addition, they presented L 2 -SP-Fisher, which further weighs each
L 2 -distance using Fisher information matrix estimated from the source dataset.
Instead of constraining the parameter search space,
Li et al.
(2019) showed that it is often more eﬀective to regularize feature maps during ﬁne-tuning, and further learns which features to pay attention to.
Learning without Forgetting (Li & Hoiem, 2018) learns to adapt the model to new tasks, while trying to match the output response on the original task of the original model using knowledge distillation (KD) (Hinton et al., 2014).
Methods proposed by Zagoruyko & Komodakis (2017) and Yim et al.
(2017) transfer knowledge from a teacher model to a student by regularizing features.
The former computes and regularizes spatial statistics across all feature maps channels, whereas the latter estimates the ﬂow of information across layers for each pair of channels, and transfers this knowledge to the student.
Instead of manually deciding the regularization penalties and what to regularize as in the previous approaches, Jang et al.
(2019) used meta-learning to automatically learn what knowledge to transfer from the teacher and to where in the student model.
Inspired by Li et al.
and Jang et al.
(2019), this paper introduces attentive feature distillation (AFD), which similarly transfers knowledge by learning from the teacher’s feature maps.
It however diﬀers from Jang et al.
(2019) as the teacher and student models share the same network topology, and it instead learns which channel to transfer from the teacher to the student in the same convolutional output.
2. Structured Sparsity
Sparsity in neural networks has been a long-studied subject (Reed, 1993; LeCun et al., 1990; Chauvin, 1989; Mozer & Smolensky, 1989;
Hassibi et al., 1994).
Related techniques have been applied to modern deep CNNs with great success (Guo et al., 2016; Dong et al., 2017a), signiﬁcantly lowering their storage requirements.
In general, as these methods zero out individual weights, producing irregular sparse connections, which cannot be eﬃciently exploited by GPUs to speed up computation.
For this, many recent work turned their attention to structured sparsity (Alvarez & Salzmann, 2016; Wen et al., 2016; Liu et al., 2017;
He et al., 2017; 2018).
This approach aims to ﬁnd coarse-grained sparsity and preserves dense structures, thus allowing conventional
GPUs to compute them eﬃciently.
Alvarez & Salzmann (2016) and Wen et al.
both added group Lasso to penalize non-zero weights, and removed channels entirely that have been reduced to zero.
Liu et al.
(2017) proposed network slimming (NS), which adds L 1 regularization to the trainable channel-wise scaling parameters γ used in batch normalization, and gradually prunes channels with small γ values by threshold.
He et al.
(2018) introduced soft ﬁlter pruning (SFP), which iteratively ﬁne-tunes and sets channels with small L 2 -norms to zero.
Pruning algorithms remove weights or neurons from the network.
The network may therefore lose its ability to process some diﬃcult inputs correctly, as the neurons responsible for them are permanently discarded.
Gao et al.
(2019) have found empirically that task accuracies degrades considerably when most of the computation are removed from the network, and introduced feature boosting and suppression (FBS).
Instead of removing neurons permanently from the network, FBS learns to dynamically prune unimportant channels, depending on the current input image.
In this paper, attentive feature selection (AFS) builds on top of the advantages of both static and dynamic pruning algorithms.
AFS not only preserves neurons that are important to some input images, but also removes unimportant ones for most inputs from the network, reducing both the memory and compute requirements for inference.
There are methods that dynamically select which paths to evaluate in a network dependent on the input (Figurnov et al., 2017;
Dong et al., 2017b; Bolukbasi et al., 2017; Lin et al., 2017;
Shazeer et al., 2017; Wu et al., 2018;
Ren et al., 2018).
They however introduce architectural and/or training method changes, and thus cannot be applied directly on existing popular models pre-trained on ImageNet (Deng et al., 2009).
Attentive Feature Distillation and Selection 3. High-Level Overview
[Figure]
We begin by providing a high-level overview of attentive feature distillation and selection (AFDS).
AFDS introduces two new components to augment each conventional batchnormalized convolutional (ConvBN) layer (Ioﬀe & Szegedy, 2015), as illustrated in Figure 2.
The AFS preemptively learns the importance of each channel, in the output of the ConvBN layer, and can suppress unimportant channels, thus allowing the expensive convolution op- eration to skip evaluating these channels.
The AFD learns the importance of each channel in the output activation, and use the importance as weights to regularize feature maps in the target model with L 2 -distance.
Each component is a small neural network containing a small number of parameters that can be trained with conventional stochastic gradient descent (SGD).
3. Preliminaries
Consider a set of training data D where each sample ( x , y ) consists of an input image x ∈ R C × H × W , and a ground-truth label y ∈ N .
Here C , H and W respectively denote the number of channels, and the height and width of the input image.
Training a deep CNN classiﬁer thus minimizes the following loss function with an optimization method based on
SGD:
[Equation]
where θ comprises all parameters of the model, the loss L CE ( f ( x , θ ) , y ) denotes the crossentropy loss between the CNN output f ( x , θ ) and the label y .
The regularizer R ( θ , x ) is often used to reduce the risk of overﬁtting.
In conventional training, R ( θ , x ) = 0.
Finally, we impose a L 2 penalty on θ , where (cid:107) z (cid:107) 2 represents the L 2 -norm of z across all its elements.
We assume that f ( x , θ ) is a feed-forward CNN composed of N ConvBN layers for feature extraction, f l ( x l − 1 , θ l ) with l ∈ L = { 1 , 2 , . . . , N } , and a ﬁnal fully-connected layer for classiﬁcation, g ( x N , θ g ).
Here, for the l th layer, x l − 1 is the input to the layer, with xindicating x , and θ l is the layer’s parameters.
Therefore, the l th layer is deﬁned as:
[Equation]
where x l ∈ R C l × H l × W l contains C l feature maps of the layer, each with a H l height and W l width.
The function conv( x l − 1 , θ l ) is a convolution that takes x l − 1 as input and uses trainable parameters θ l , and norm( z ) performs batch normalization.
Finally, γ l , β l ∈ R C l are trainable vectors, the multiplications ( · ) and additions (+) are channel-wise, and relu( z ) =
max( z , 0) stands for the ReLU activation.
Although we use the feed-forward classiﬁer above for simplicity, it can be easily modiﬁed to contain additional structures such as residual connections (He et al., 2016) and computations for object detection (Ren et al., 2015).
During transfer learning, as we ﬁne-tune the network with a diﬀerent task, the ﬁnal layer g ( x N , θ g ) is generally replaced with a new randomly-initialized one h ( x N , θ h ).
To prevent overﬁtting, additional terms are used during transfer learning, for instance, L 2 -SP (Li et al., 2018) further constrains the parameters θ l to explore around their initial values θ (cid:63)l :
[Equation]
Instead of regularizing parameters, methods based on knowledge distillation (Hinton et al., 2014) encourages the model to mimic the behavior of the original while learning the target task.
Learning without Forgetting (LwF) (Li & Hoiem, 2018) uses the following regularizer to mimic the response from the original classiﬁers:
[Equation]
where f L ( x , θ L ) indicates the ﬁrst N layers, and g (cid:63) and θ (cid:63)g respectively denote the original fully-connected (FC) layer and its associated parameters, and generally λ LwF = 1.
Zagoruyko & Komodakis (2017), Yim et al.
(2017) and Li et al.
(2019) chose to regularize feature maps in some intermediate layers L (cid:48) ⊆
L .
We assume that x (cid:63)l is the l th layer output of the original model with weights θ (cid:63) when the input x is shown to the model, and r is a method-dependent function that constrains the relationship between x (cid:63)l and x l .
The regularizer can then be deﬁned as follows:
[Equation]
3. Attentive Feature Distillation
A simple way to extend Equation (5) is to constrain the L 2 -norm-distance between x (cid:63)l and x l , and thus pushing the target model to learn the feature map responses of the source:
[Equation]
The above formulation, however, places equal weight to each channel neurons of the feature maps.
As we discussed earlier, the importance of channel neurons varies drastically when diﬀerent input images are shown.
it is thus desirable to enforce a diﬀerent penalty for each channel depending on the input x .
For this purpose, we design the regularizer:
[Equation]
Note that in Equation (7), for any tensor z , the term z [ c ] denotes the c th slice of the tensor.
The transfer importance predictor ρ l : R C l × H l × W l → R C l computes for each channel the importance of the source activation maps, which governs the strength of the L 2 regularization for each channel.
The predictor function is trainable and is deﬁned as a small network with two FC layers:
[Equation]
The function (cid:91) : R C × H × W
→ R C × HW ﬂattens the spatial dimensions in a channel-wise fashion; The parameters ϕ l ∈ R HW × H , ν l ∈ R 1 × H , ϕ (cid:48) l ∈ R H and ν (cid:48) l ∈ R C can thus be trained to adjust the importance of each channel dynamically; ﬁnally, the softmax activation is borrowed from attention mechanism (Vaswani et al., 2017) to normalize the importance values.
In our experiments, ϕ l and ϕ (cid:48) l use He et al.
(2015)’s initialization, ν l and ν (cid:48)
l are both initialized to 0 .
3. Attentive Feature Selection
In a fashion similar to feature boosting and suppression (FBS) (Gao et al., 2019), AFS modiﬁes the ConvBN layers from Equation (2):
[Equation]
where the predictor function takes as input the activation maps of the previous layer, i.e. π l : R C l − 1 × H l − 1 × W l − 1 → R C , is used to replace the vector γ l .
This function dynamically predicts the importance of each channel, and suppresses certain unimportant channels by setting them to zero.
The expensive conv function can hence be accelerated by skipping the disabled output channels.
The predictor function is deﬁned as below:
[Equation]
where m l , s l ∈ { 0 , 1 } C l are both constant masks that take binary values: m l prunes output channels by permanently setting them to zeros, and s l decides for each channel whether the output of h l ( x l − 1 ) or γ l should be used.
It is clear that when m l = 1 , no channel neurons are removed from the network.
In Section 3.5, we explain how m l and γ l can be determined during the ﬁne-tuning process.
The winner-take-all function wta
[Equation]
( z ) preserves the (cid:100)
dC l (cid:101) most salient values in z , and suppresses the remaining ones by setting them to zeros.
The density value 0 < d ≤ 1 is a constant that controls the number of channels to preserve during inference, with 1 preserving all C l channels.
The smaller d gets, the more channels can be skipped, which in turn accelerates the model.
Finally, the function h l : R C l − 1 × H × W
→ R C l is a small network that is used to predict the importance of each channel.
It is composed of a global average pool followed by a FC layer, where pool : R C l − 1 × H × W
→ R C l − 1 computes the average across the spatial dimensions for each channel:
[Equation]
For the initialization of the FC parameters, we apply He et al.
(2015)’s method on the trainable weights ϕ (cid:48)(cid:48)
l ∈ R C l − 1 × C l and ν (cid:48)(cid:48)
l ∈ R C l is initialized to zeros.
3. Training Procedure
In this section, we describe the pipeline of AFDS for transferring knowledge from a source model to a new model by ﬁne-tuning on target dataset.
The detailed algorithm can be found in Appendix A.
Initially, we have a pre-trained model f with parameters θ (cid:63) for the source dataset ( e.g. ImageNet).
To ensure better accuracies on compressed target models, All ConvBN layers f l in f are extended with AFS as discussed in Section 3.4, with d initially set to 1, which means that all output channels in a convolutional layer are evaluated during inference, i.e. no acceleration.
The pre-trained model is then ﬁne-tuned on the target training dataset D with the AFD regularization proposed in Section 3.3.
Empirically we found that in residual networks with greater depths, AFS could become notably challenging to train to high accuracies.
To mitigate this, for each output channel of a layer l we update s l according to the variance of h l ( x l − 1 ) observed on the target dataset.
For each channel if the variance is smaller than a threshold δ s , then we set the entry in s l to zero for that particular channel.
This action replaces the output of h l ( x l − 1 ) with γ l , which is a trainable parameter initialized to the mean of h l ( x l − 1 ).
We compute the mean and variance statistics using Welford (1962)’s online algorithm which can eﬃciently compute the statistics in a single-pass with O (1) storage.
In our experiments, δ s is set to a value such that 50% of the channel neurons use the predictor function h l .
Moreover, we discovered that many of the channel neurons are rarely activated in a AFSbased network.
We further propose to remove the channel neurons that are activated with a low frequency.
In each layer l , the mask m l is used to disable certain channels from the network by setting their output to a constant 0 , if the probability of a channel neuron being active is lower than δ m .
Zeroed-out channels can thus be permanently removed when the model is used in inference.
Experiments
In this section we provide an extensive empirical study of the joint methods of transfer learning and channel pruning.
We evaluate the methods with 6 diﬀerent benchmark datasets:
Caltech-256 (Griﬃn et al., 2007) of 256 general object categories; Stanford Dogs 120 (Khosla et al., 2011) specializes to images containing dogs;
MIT Indoors 67 (Quattoni & Torralba, 2009) for indoor scene classiﬁcation; Caltech-UCSD Birds-200-2011 (CUB-200-2011)
(Wah et al., 2011) for classifying birds; and Food-101 (Bossard et al., 2014) for food categories.
We refer to Li et al.
(2018) and Li et al.
(2019), for a detailed description of the benchmark datasets.
For Caltech-256, we randomly sample either 30 or 60 images from the training set for each category to produce Caltech-256-30 and -60 training datasets.
We use the ResNet-101 from torchvision 1 pre-trained on ImageNet as the network for experiments.
For ResNet-101 equipped with AFS, we start by extending the pre-trained model and replacing each batch normalization with a randomly initialized AFS, and ﬁne-tune the resulting model on ImageNet for 90 epochs with a learning rate of 0.01 decaying by a factor of 10 every 30 epochs.
The resulting model matches its original baseline accuracy.
For each benchmark dataset, the ﬁnal FC layer of the network is replaced with a new
FC randomly initialized with He et al.
(2015)’s method to match the number of output categories accordingly.
We then perform transfer learning with 4 diﬀerent methods: L(ﬁne-tuning without additional regularization), L 2 -SP (Li et al., 2018), learning without forgetting (LwF) (Li & Hoiem, 2018), and ﬁnally AFD for models using AFS.
To accelerate the resulting ﬁne-tuned models, we continue ﬁne-tuning the model while gradually pruning away channels used during inference.
For this, we separately examine 3 pruning strategies: network slimming (NS) (Liu et al., 2017), soft ﬁlter pruning (SFP) (He et al., 2018) and ﬁnally AFS for models transfer learned with AFD.
Note that NS prunes channels by sorting them globally, while SFP does so in a layer-wise manner with identical prune ratios.
During this procedure, we start with an unpruned model and incrementally remove 10% of the channels used in inference,
i
[Table]
e. preserving 90%, 80%, and etc. , down to 10% of all channels for the accelerated models.
At each step, we ﬁne-tune each model usingsteps of SGD with a batch size of 48, at a learning rate of 0.01, before ﬁne-tuning for a further 4500 steps at a learning rate of 0.001.
AFS additionally updates the m and s masks between the two ﬁne-tuning runs.
[Table] [Table]
For each pruned model, we can compute the number of multiply-accumulate operations (MACs) required to perform inference on an image.
For each accelerated convolution, the required number of MACs is k 2 HWC in C out , where C in and C out are the number of input and output channels that are not pruned, respectively.
We compute the total number of
MACs by summing up the MACs in all convolutions, residual connections, and the ﬁnal pooling and FC layers.
For AFS as we dynamically select which channels to evaluate during inference, we additionally add the overhead of the importance predictor layers to the number of total MACs.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In Figure 3, we present the trade-oﬀ relationship between the number of vs. the target dataset accuracies for Stanford Dogs and Caltech-256-60.
It is clear that AFDS (ours) exceeds various combinations of pruning methods (NS, SFP) and transfer learning methods ( L 2 , L 2 -SP, LwF).
The results for the remaining datasets can be found in Appendix B.
The trade-oﬀ curves show that AFDS minimizes accuracy degradation even if 47% of the total MACs are removed from the original model, AFDS resulted in only 1 . 83% drop in accuracy for the model trained on Stanford Dogs.
In extreme cases where we permit only 110 of the original computations, our method can still manage a 70 .
70% accuracy, which is substantially better when compared to other pruning algorithms: NS drops to 1 .
33% and
SFP only has 59 .
Table 1 provide numerical comparisons of diﬀerent pruning methods against AFS under various speed-up constraints.
Table 2 similarly compares transfer learning strategies against
AFD.
Under most acceleration requirements, the combined method, AFDS, achieves the best accuracies on the target datasets.
Finally, Table 3 compares AFDS against other literatures that performs transfer learning.
AFDS can achieve state-of-the-art accuracies when compared to methods that produce models with similar number of MACs.
Conclusion
In this paper, we introduced attentive feature distillation and selection (AFDS), a dualattention method that aims to reap the advantages of transfer learning and channel pruning methods.
By applying AFDS during ﬁne-tuning, we can not only learn a new model with a higher target task accuracy, but also further accelerates it by computing a subset of channel neurons in each convolutional layers.
Under a wide range of datasets, we demonstrated the smallest drop in validation accuracies under the same speed-up constraints when compared to traditional compression methods such as network slimming (Liu et al., 2017) and soft ﬁlter pruning
He et al., 2018).
Acknowledgements
This work is supported in part by National Key R&D Program of China (No.
2019YFB2102100), Science and Technology Development Fund of Macao S.A.R (FDCT) under number 0015/2019/AKP, Shenzhen Discipline Construction Project for Urban Computing and Data Intelligence, and the National Natural Science Foundation of China (No.
A The Overall Training Algorithm
In Algorithm 1 we illustrate the complete training procedure described above.
Here, the function takes as input the target training dataset D , the source model f and its parameters θ (cid:63) , the total number of steps to ﬁne-tune S , the initial learning rate α and the threshold hyperparameters δ s and δ m respectively for s l and m l .
The function returns the optimized parameters θ for the target dataset, and both constant masks for all layers s =
( s 1 , s 2 , . .
, s L ) and m = ( m 1 , m 2 , . . .
, m L ).
The function SGD then ﬁne-tunes the model parameters.
For each layer l , we compute the mean µ l and variance σ l statistics of q l ( x l − 1 ), and use it to compute s l .
1: function AFDS ( D , f, θ (cid:63) , S, α, δ s , δ m ) 2: for l ∈ L : s l ←3: for l ∈ L : m l ←4: θ ← SGD( D , f, θ (cid:63) , s , m , (cid:100) S 2 (cid:101) , α, R ) 5: for l ∈ L do 6: µ l ← E ( x ,y )
∼D [ q l ( x l − 1 )] 7: σ 2 l ← E ( x ,y )
∼D
[( q l ( x l − 1 ) − µ l ) 2 ] 8: p l ← E ( x ,y )
∼D [ π l ( x l − 1 ) > 0] 9: s l ← σ 2 l > δ s 10: γ l ← µ l 11: m l ← p l > δ m 12: end for 13: θ ← SGD( D , f, θ , s , m , (cid:100) S 2 (cid:101) , α 10 , R ) 14: return θ , s , m 15: end function
B Additional Results
[Figure]