N ETWORK COMPRESSION USING C ORRELATION A NALYSIS OF L AYER R ESPONSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression.
PFA exploits the intrinsic correlation between ﬁlter responses within network layers to recommend a smaller network footprint.
We propose two compression algorithms: the ﬁrst allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer.
Both algorithms are evaluated against several architectures and datasets and we show considerable compression rates without compromising accuracy, e.g., for VGGon CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively.
In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio.
Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.
I NTRODUCTION
Despite decades of research, the design of neural networks is still an empirical process.
Practitioners make design choices, such as the number of layers, type of layers, number of ﬁlters per layer, etc., based on intuition or brute-force search.
Nevertheless, the performance of these algorithms, together with the advances of GPU devices, have led to a growing popularity of these techniques in both academia and industry.
Recent advances are unveiling some properties of neural networks.
For example, there is a consensus that depth can accelerate learning, and that wider layers help optimization (Arora et al., 2018;
Livni et al., 2014; Cohen et al., 2018).
However, in practical applications, the size of these networks is often a limiting factor when deploying on devices with constrained storage, memory, and computation resources.
Another known neural network property is that the responses of a layer exhibit considerable correlation (Denil et al., 2013), inspiring the idea of learning decorrelated ﬁlters (Cogswell et al., 2016;
Rodríguez et al., 2017).
These algorithms propose a modiﬁed loss function to encourage decorrelation during training and show that accuracy improves with decorrelated ﬁlters.
However, such algorithms focus on training and do not address network compression.
Our hypothesis is that layers that exhibit high correlation in ﬁlter responses could learn equally well using a smaller number of ﬁlters.
Principal Filter Analysis (PFA) draws from the recent ﬁndings that it is easier to start with an overparametrized network and it then exploits intra-layer correlation for guiding network compression.
PFA analyzes a trained network and is agnostic to the training methodology and the loss function.
Inference is performed on a dataset, and the correlation within the responses of each layer is used to provide a compression recipe.
A new smaller architecture based on this recipe can then be retrained.
We propose two closed-form algorithms based on spectral energy analysis for suggesting the number of ﬁlters to remove in a layer:
PFA-En uses Principal Component Analysis (PCA) (Hotelling, 1933) to allow a user to specify the proportion of the energy in the original response that should be preserved in each layer; PFA-KL is a heuristic that leads to a parameter-free approach that uses Kullback-Leibler (KL) divergence (Kullback & Leibler, 1951) to identify the number of redundant ﬁlters.
Based on the new suggested number of ﬁlters per layer identiﬁed by PFA, we remove those that are maximally correlated with other ﬁlters and ﬁne-tune the network by retraining.
Both PFA algorithms are straightforward to implement and, as shown in Sec.
4, they achieve better compression and, in most cases, better accuracy than the state of the art on several datasets and architectures.
In Sec.we also show how PFA can be used to perform simulations compression and domain adaptation.
R ELATED WORK
The ﬁeld of network compression encompasses a wide range of techniques that can be grouped into the following families: quantization, knowledge distillation, tensor factorization and network pruning.
Quantization algorithms compress networks by reducing the number of bits used to represent each weight (Wu et al., 2016; Han et al., 2016a; Rastegari et al., 2016;
He & Cheng, 2018; Wu et al., 2018).
Knowledge distillation (Hinton et al., 2014) and model compression (Buciluˇa et al., 2006) aim to create a simpler model that mimics the output of a more complex model.
Variations on this concept include Ba & Caurana (2014); Romero et al.
; Chen et al.
Tensor factorization algorithms exploit the redundancy present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses (Denton et al., 2014; Lebedev et al., 2014; Jaderberg et al., 2014; Cheng et al., 2016;
Masana et al., 2017; Wen et al., 2017; Yu et al., 2017; Alvarez & Salzmann, 2017; Peng et al., 2018).
Network pruning is a family of techniques that compress networks by iteratively removing connections based on the salience of their weights.
Early work, like Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi et al., 1993), targeted fully connected networks.
Recent work can be divided into two sub-families: sparse pruning (Han et al., 2016b; Srinivas & Babu, 2015; Wen et al., 2016; Tung & Mori, 2018;
Aghasi et al., 2017; Carreira-Perpinan & Idelbayev, 2018; Zhang et al., 2018; Dai et al., 2018), where individual neurons are removed, and structured pruning (Li et al., 2017; He et al., 2017; Luo et al., 2017; Molchanov et al., 2017; Yu et al., 2018), where entire ﬁlters are removed.
PFA falls within the family of structured network pruning.
Some of these techniques, e.g., Li et al.
(2017), require user deﬁned parameters that are hard to choose and whose effect on the footprint is difﬁcult to predict (see Sec. 4.6 for a more detailed discussion).
Others also require modiﬁcation of the loss function, e.g., Liu et al.
In contrast, PFA-En has only one intuitive parameter, which is the proportion of the response energy to be preserved at each layer, and PFA-KL is parameter-free.
Furthermore, instead of learning the saliency of the ﬁlters during training by modifying the loss function, PFA estimates it after training without requiring knowledge of the training details.
This makes PFA applicable to any trained network, without the need to know its loss function.
Within the structured pruning family, there are approaches based on the singular value decomposition (SVD) (Xue et al., 2013; Nakkiran et al., 2015;
Prabhavalkar et al., 2016), where a new set of ﬁlters are obtained by projecting the original weights onto a lower dimensional space.
PFA differs from these methods by the fact that SVD is performed on the responses of the layers rather than on the ﬁlter weights, and no projection is done.
This is particularly relevant for domain adaption applications, where a trained network is specialized for a different task.
Techniques that make compression decisions based on weights , rather than the responses , cannot take into account the speciﬁcity of the task.
As shown in Sec. 4.7, PFA derives different architectures from the same initial model when analyzed for different tasks.
Some methods, e.g., Li et al.
(2017); Masana et al.
; Peng et al.
(2018), also reason on the layer responses.
These techniques aim to ﬁnd a smaller set of ﬁlters that, in different ways, minimize the reconstruction error of the feature maps or the response output.
Note that PFA uses the spectral energy of the ﬁlters’ responses only to decide how many ﬁlters should be preserved.
Similarly, the subsequent ﬁlter selection process does not take the reconstruction error into account.
PFA is also philosophically and practically different: PFA uses the concept of correlation within the responses to identify redundancy within a layer.
In practice this means that PFA can compress all layers in one run, while the majority of the techniques that use responses need to operate on one layer at the time.
Finally, PFA is orthogonal to the quantization, tensor factorization and distillation methods, and could be used as a complementary strategy to further compress neural networks.
P RINCIPAL F ILTER A NALYSIS
In this section, PFA-En and PFA-KL algorithms are described in detail.
Both algorithms share the idea of exploiting correlations between responses in convolutional layers and neurons in fully connected layers to obtain a principled recommendation for network compression.
3. D EFINITIONS
PFA is inherently data driven and thus takes advantage of a dataset { X i } ∈ R M × I where X i is the i th input data sample, M is the number of samples in the dataset, and I is the input dimensionality.
Typically, this dataset is the data used to train the network, but it can also be a representative set that covers the distribution of inputs likely to be encountered.
Without loss of generality, we assume that the input data are images: { X i } ∈ R M
×
H
×
W × C , where H is the image height
, W is the image width and C is the number channels.
Let T [ (cid:96) ] i ∈ R 1 × H
[ (cid:96) ]
× W [ (cid:96) ] × C
[ (cid:96) ] be the output tensor produced by a given layer (cid:96) of a network on the i th input sample.
Any operation in the network is considered a layer (e.g., batch normalization, ReLU, etc.).
In this work we analyze the output of convolutional and fully connected layers.
However, PFA can be used to analyze the output of any layer in the network.
Recurrent neural networks could also be analyzed, but are out of the scope for this paper.
For a convolutional layer (cid:96) , let
W [ (cid:96) ] ∈
R
f
[ (cid:96) ] h × f
[ (cid:96) ] w × C
[ (cid:96) − 1] × C [ (cid:96) ] be the set of C [ (cid:96) ] trainable ﬁlters with a kernel of size f
[ (cid:96) ] h × f
[ (cid:96) ] w × C
[ (cid:96) − 1] .
Therefore, we can formally express T [ (cid:96) ]
i produced by the convolutional layer as T [ (cid:96) ] i =
W
[ (cid:96) ] ∗ T [ (cid:96) − 1] i with T [0] i
= X i , where ∗ denotes the convolution operator.
We omit the bias term to improve readability.
We deﬁne the response vector a [ (cid:96) ] i ∈ R C
[ (cid:96) ] of a given layer (cid:96) with respect to an input X
i to be the spatially max-pooled and ﬂattened tensor T [ (cid:96) ]
i (i.e., max-pooling over the dimensions H [ (cid:96) ] and
W [ (cid:96) ] ).
For fully-connected layers, W [ (cid:96) ] ∈ R C
[ (cid:96) ] , with C [ (cid:96) ] being the number of neurons in layer (cid:96) .
The output tensor is T [ (cid:96) ]
i =
W
[ (cid:96) ]
T [ (cid:96) − 1] i , and since no pooling is required, the response vector is a [ (cid:96) ]
i = T [ (cid:96) ] i ∈ R C
[ (cid:96) ] .
Let A [ (cid:96) ]
=
[ a [ (cid:96) ] 1 , . . .
, a [ (cid:96) ]
M ] (cid:62) ∈ R M × C [ (cid:96) ] be the matrix of responses of layer (cid:96) given a dataset with M samples.
Lastly, let λ [ (cid:96) ]
∈ R C
[ (cid:96) ] be the distribution of eigenvalues of the covariance matrix of A [ (cid:96) ] sorted in descending order and normalized to sum to 1 .
In the following sections we present two algorithms that exploit the distribution λ
[ (cid:96) ] to guide network compression.
3. C OMPRESSION RECIPES
The distribution λ [ (cid:96) ] provides insight into the correlation within layer (cid:96) .
The closer λ
[ (cid:96) ] is to a uniform distribution, the more decorrelated the response of the ﬁlters and the more uniform their contribution to the overall response energy.
Conversely, the closer λ [ (cid:96) ] is to a Dirac δ -distribution, the more correlated the ﬁlters.
Our hypothesis is that layers that exhibit high correlation in ﬁlter responses could learn equally well using a smaller number of ﬁlters.
Now that we have deﬁned the key ingredient ( λ [ (cid:96) ] ) for PFA, we present two strategies that produce a compression recipe with the goal of maximizing compression while reducing correlation.
Let a compression recipe Γ = { γ [ (cid:96) ] } , with γ [ (cid:96) ] ∈ (0 , 1] , be the set of compression factors applied to each of the L layers included in the analysis.
For example, γ [3] = 0 .
6 means that we keep 60% of the ﬁlters in layer 3.
Once the correct number of ﬁlters have been determined by the recipe, one could further proceed to choose which ﬁlters should be kept.
We call this ﬁlter selection and we outline it in Sec.
3.2. PFA-E N : ENERGY - BASED RECIPE
PCA can be used for dimensionality reduction by performing a linear mapping to a lower dimensional space that maximizes the variance of the data in this space.
This can be accomplished by extracting the eigenvectors and eigenvalues of the covariance matrix.
The original data are then reconstructed using the minimum number of eigenvectors that correspond to the eigenvalues that sum up to the desired energy factor τ .
Inspired by this strategy, we propose to keep the minimum set of ﬁlters such that a fraction of response energy greater or equal to a user deﬁned τ is preserved.
We deﬁne the energy at a given compression ratio for a layer as
[Equation]
and we propose to re-architect the network according to the following recipe:
[Equation]
The parameter τ provides the user with the ability to guide the compression ratio.
PFA-En has the advantage of being tightly connected to well-established dimensionality reduction techniques based on PCA, it is simple to implement and uses a single, highly intuitive parameter.
Furthermore, since evaluating the size of a model (or its FLOPs) obtained at different energy thresholds is easy and fast, it is straightforward to replace the parameter τ with the desired footprint (or FLOPs) after compression.
Being able to specify a target footprint instead of an energy threshold gives PFA-En even more appeal for practical use cases.
3.2. PFA-KL: KL DIVERGENCE - BASED RECIPE
We propose an alternative formulation to obtain an optimal recipe Γ (cid:63) KL , based on KL divergence.
This formulation is a heuristic that frees PFA from the use of any parameter.
As previously mentioned, a distribution λ [ (cid:96) ] similar to a ﬂat distribution implies an uncorrelated response of the ﬁlters in layer (cid:96) .
Therefore, the farther the distribution λ [ (cid:96) ] is from a ﬂat distribution the more layer (cid:96) can be compressed.
Let us deﬁne u as the desired uniform distribution (no correlation between ﬁlters), and d = Dirac ( k ) as the worst case distribution (all ﬁlters are perfectly correlated).
We can measure the dissimilarity of the actual distribution, λ [ (cid:96) ] , from the desired distribution, u , as the KL divergence KL( λ [ (cid:96) ] , u ) .
The upper bound of which is given by u KL = KL( d , u ) , while the lower bound is 0.
Notice that one could replace the KL divergence with any dissimalarity measure between distributions, such as χ 2 or the earth mover’s distance (Rubner et al., 1998).
Intuitively, when the actual distribution is identical to the ideal distribution (i.e., no correlation found) then we would like to preserve all ﬁlters.
On the other hand, when the actual distribution is identical to the worst case distribution (i.e., all ﬁlters are maximally correlated) then one single ﬁlter would be sufﬁcient.
The proposed KL divergence-based recipe is a mapping ψ : [0 , u KL ] (cid:55)→ (0 , 1] ; a divergence close to the upper bound results in a strong compression and a divergence close to the lower bound results in a milder compression:
[Equation]
In this work, we use a simple linear mapping ψ ( x, u KL )
= 1 − x / u KL .
Other mappings were explored, leading to different degrees of compression; however, we have empirically observed that a linear mapping produces good results that generalize well across networks.
3.2. F ILTER SELECTION
The recipes produced by PFA-En and PFA-KL provide the number of ﬁlters, F [ (cid:96) ]
= (cid:6) γ
[ (cid:96) ] · C [ (cid:96) ] (cid:7) , that should be kept in each layer, but do not indicate which ﬁlters should be kept.
One option is to retrain the compressed network from a random initialization ( from scratch ).
In this case it does not matter which ﬁlters are chosen.
An alternative is to select which ﬁlters to keep and use their values for initialization.
We do this by removing those ﬁlters in each layer that are maximally correlated.
We compute the (cid:96) 1 -norm of each row of the correlation matrix from the matrix
A
[ (cid:96) ] and remove the ﬁlter with the largest norm.
If more ﬁlters need to be chosen then we update the correlation matrix by removing the previously selected ﬁlter, and iterate until the desired number of ﬁlters has been removed.
In the rare, but theoretically possible, case in which two ﬁlters have the same (cid:96) 1 -norm we choose the one with the highest individual correlation coefﬁcient.
E XPERIMENTS
To evaluate PFA, we analyze several network architectures and datasets, and our results are compared to the state of the art.
Speciﬁcally we will contrast PFA against the ﬁlter pruning approach (FP) in Li et al.
(2017), the network slimming approach (NS) in Liu et al.
(2017), the variational information bottleneck approach (VIB) in Dai et al.
and the ﬁlter group approximation approach (FGA) in Peng et al.
For comparison, we focus on the compression ratio and the accuracy change, measured in percentage points (pp), obtained by the compressed architectures.
This enables plotting various techniques in the same plot, even if the accuracy of each original architecture is slightly different because of different training strategies used.
After the Sate of the Art quantitative comparison, PFA is tested on the task of simultaneous compression and domain adaptation.
4. N ETWORKS AND DATASETS
Three networks are evaluated in this paper: VGG-16 (Simonyan & Zisserman, 2015) (we use the version proposed by Zagoruyko (2015) for CIFAR), ResNet-56 (He et al., 2016), and SimpleCNN, a small CNN we use to demonstrate the capabilities of PFA.
Refer to Appendix A for the architecture details.
We train and test all the networks on CIFAR-10 and CIFAR-100 (Krizhevsky, 2009).
In addition we test VGG-16 (Long et al., 2015) on ImageNet (Deng et al., 2009).
4. M ETHODOLOGY
A baseline for each architecture is obtained by training using 10 random initializations – we choose the initialization that leads to the highest test accuracy and perform inference on the training set to obtain the responses at each layer ( A [ (cid:96) ] ) needed for the PFA analysis.
PFA analyzes all layers in parallel to obtain its recipes.
Compressed architectures are created by following the recipes from each PFA strategy.
For each new compressed architecture, we perform two types of training.
First, we retrain with 10 different random initializations and this set of results are referred to as Scratch .
Second, we retrain 10 times using the ﬁlter selection method described in Sec.
3.2.3, and ﬁne-tune the compressed network starting from the weights of the selected ﬁlters.
The accuracy reported is the mean of these 10 retrainings.
Note that the retraining is done without hyper-parameter tuning.
While this is a sub-optimal strategy, it removes any ambiguity on how well the parameters were tuned for the baseline compared to the compressed networks.
In practice, one would expect to attain even better results if parameter sweeping was done on the compressed networks.
In all experiments we use the SGD optimizer with no decay, learning rate 0.1, Nesterov momentum 0.9, 50 epochs for SimpleCNN and 160 for VGG-16 and ResNet-56.
PFA-En is computed for every energy value: τ ∈ { 0 . 8 , 0 . 85 , 0 . 9 , 0 . 925 , 0 .
98 , 0 . 99 } , whereas PFA-KL is parameter-free and is computed once per baseline network.
4. E MPIRICAL U PPER -B OUND FOR S IMPLE CNN ON CIFAR-10 AND CIFAR-
An empirical upper bound on the accuracy at different compression ratios is obtained by randomly choosing how many ﬁlters to remove at each layer.
By repeating this a sufﬁcient number of times, the best result at each compression ratio can be considered an empirical upper bound for that architecture and footprint, labeled Upper bound .
On the other hand, the result averaged across all trials is representative of how easy (or difﬁcult)
it is to randomly compress a network without hurting its accuracy, labeled Avg.
random .
Results tagged with the word Scratch were trained from a random initialization while those without the tag were obtained using ﬁlter selection.
In these experiments we generated and trained 300 randomly pruned architectures for each compression ratio.
In Fig.
1 the accuracy drops for Avg.
random are particularly severe, showing that in general random pruning is not an effective strategy, while Upper bound shows excellent results.
On CIFAR-10, both PFA strategies trained from scratch produce results close to Upper bound Scratch and signiﬁcantly better than the Avg.
random Scratch results.
This indicates that choosing the correct number of ﬁlters for each layer is crucial even for footprints that are close to the original model, and shows that PFA is able to perform this choice close to optimally.
PFA-En Scratch and PFA-KL
Scratch both achieve a footprint close to 60% with an accuracy drop of only 0.5 pp.
When performing ﬁlter selection, rather than retraining from scratch, both PFA-En and PFA-KL improve their accuracy by about 1 pp above the Scratch version for footprints above 40%.
For footprints above 42%, the ﬁlter selection strategy performs even better than Upper bound Scratch , whereas for footprints less than 40%, the accuracy starts to degrade.
On CIFAR-100, ﬁlter selection increases its gain up to 2 pp compared to the Scratch version, despite the more challenging nature of the dataset as indicated by the faster drop in accuracy past the 40% footprint mark.
We have found that the ﬁlter selection strategy converges faster during training and performs consistently better across different architectures and datasets, hence from now on we will only report results using PFA with ﬁlter selection.
Interestingly, at the 10% footprint mark a random initialization appears to be better than the use of ﬁlter selection.
It is possible that when keeping an extremely small number of ﬁlters, the starting point provided by the ﬁlter selection becomes a local minimum that is difﬁcult to escape.
For thin layers in relatively small architectures (like SimpleCNN), a random initialization may give more room for exploration during the learning phase.
4. VGG-16 AND R ES N ET -56 ON C IFAR -10, C IFAR -100 AND I MAGE N ET
For VGG-16 we compare the results of PFA with those reported by FP, NS, VIB 1 and FGA 2 , after a single iteration of NS for a direct comparison with the other methods (see Fig.
2a, 2b and Tab. 1).
Note that VIB performs sparse pruning (single neurons) rather than structured pruning.
The main results are summarized in Tab. 1.
PFA-En and PFA-KL outperform all the other methods both on CIFAR-10 and CIFAR-100 in terms of compression ratio.
On CIFAR-10, PFA-En ( τ = 0 . 98 ) obtains a model 2.5x, 2.8x and 4.5x smaller than NS, FP and VIB respectively.
On CIFAR-100, PFA-En also obtains models 1.2x, 2x smaller than NS and VIB.
Despite achieving smaller models the accuracy change is comparable to that of the other techniques: all techniques achieve an accuracy change between -1pp and +2pp from the baseline.
Similarly, the FLOPs reduction is inline with that achieved by other techniques. Note that
[Table] [Figure] [Table]
VGG
[Table]
fully conv.
[Table]
PFA-En obtains similar model sizes as FGA on CIFAR-100, but with better accuracy for compression ratios over 13%.
At a comparable compression ratio of 40%, PFA-En achieves +1pp better accuracy than FGA.
For ResNet-56 on CIFAR-10 the same conclusions hold true.
PFA-En ( τ = 0 . 98 ) achieves an accuracy similar to FP but with a smaller footprint: 61.5% compared to 86.3%.
PFA-KL achieves an even smaller footprint: 59.6%, with an accuracy drop of only 0.6 pp.
Similar results are observed on
CIFAR-100, as shown in Fig.
B.1b in Appendix B.
For VGG-16 on ImageNet we compare PFA-KL with FGA (note that NS also reports results on
VGG-16 but with fully connected layers).
While FGA reports footprints between 74.4% and 23.18% we report in Tab. 1 the footprint most closely comparable with PFA-KL: at a slightly smaller footprint PFA-KL achieves a better accuracy improvements than FGA, increasing the accuracy over the baseline by 2.4%. In summary, independently of the architecture or dataset, PFA consistently provides better compression or accuracy than the state of the art. 4. O N THE COMPLEXITY AND SCALABILITY OF PFA
The complexity of PFA (excluding the inference step), with respect to number of ﬁlters and dataset size, is dominated by the the PCA analysis which, for a given layer, is O ( mn 2 + n 3 ) , n being the number of ﬁlters, and m the number of samples.
For example, for ImageNet (Deng et al., 2009), m =1.2M, and assuming a VGG-16 architecture with layers of size n equal to 64, 128, 256, 512, and 4096, the time to compute PFA per layer is roughly 1.24s, 2.8s, 4.6s, 9.3s, and 127.5s respectively (single CPU @ 2.30GHz).
The complexity of the ﬁlter selection only depends on the layer size.
In the worst case is O (˜ nn 2 ) , where ˜ n is the number of ﬁlters to remove.
Considering that PFA has to run once at the end of the training step, the time consumed by PFA is negligible compared to the whole training time.
In exchange for this negligible extra-time, PFA provides the long-term beneﬁt of a smaller footprint and faster inference, which, in the lifetime of a deployed network, including re-training when new data becomes available, will quickly surpass the time initially required by PFA.
4. L IMITATION OF THE STATE OF THE ART
All the techniques we used for comparison achieve good results in term of maintaining accuracy and reducing FLOPs.
In our view their main limitation, and difference with respect to PFA, is in the user deﬁned parameters that they rely upon.
NS has two crucial parameters: the weight of the sparsity regularizer, and the percentage of ﬁlters to be pruned.
The weight has a direct impact on the induced sparsity, however, there is no intuitive way to set this parameter and it needs to be tuned for each architecture and dataset.
In addition, setting the same percentage of ﬁlters to be pruned at each layer for the whole network ignores the relative effect of those ﬁlters on the accuracy and the footprint of the network.
Setting the thresholds in FP is non-trivial and it requires the user to choose the compression thresholds based on a pre-analysis that provides insight on the sensitivity of each layer to pruning.
Both VIB and FGA require a user tuned parameter that controls how much each layer is compressed.
The tuning is different depending on the network-dataset combination and depending on the layer.
From the results there seem to be no intuitive way to set this parameter other than trial and error.
In contrast, PFA achieves comparable accuracy and FLOPs with a much higher compression ratio.
In addition, one of the advantages of PFA is that it requires a single intuitive parameter (for example the desired footprint in PFA-En), or it is parameter-free (PFA-KL).
On the other hand, all the state of the art techniques we used for comparison require a tuning step.
4. S IMULTANEOUS COMPRESSION AND DOMAIN ADAPTATION USING PFA
The typical PFA use case is to compress a network for the same initial domain D A used for training.
However, PFA can be computed using data from a different domain, D Z .
The result is a compressed architecture specialized for the target domain D Z that takes advantage of the training initially performed on D A .
In this section we show that PFA derives different architectures from the same initial model when analyzed for different tasks, hence, in addition to compression PFA can also be used for domain adaptation.
In order to test PFA on domain adaptation we randomly sample classes out of the originalcontained in CIFAR-100.
We generate two sub-sets ( D R1 Z , D R2 Z ) of 10 classes each, and four sub-sets ( D S1 Z , D S2 Z , D S3 Z , D S4 Z ) of 2 classes each.
We perform experiments adapting from initial domains
D C10 A = CIFAR-10 and D C100 A = CIFAR-100 to the above mentioned target domains.
Note that all
D Z are subsets of D C100 A but not of D C10 A .
For each adaptation D
A → D Z we run the following experiments using a VGG-16 model:
• Full scratch : Train from scratch on domain D Z .
• Full ﬁne : Train from scratch on domain D A and ﬁne-tune on D Z .
• PFA scratch :
Train from scratch on domain D A , run PFA-KL on domain D Z and train compressed architecture from scratch on D Z .
[Figure]
• PFA ﬁne : Train from scratch on domain D A , run PFA-KL on domain D Z and train compressed architecture using ﬁlter selection on D Z .
The results in Fig.
3 show how the PFA ﬁne strategy performs very similarly to the full ﬁned tuned model ( Full ﬁne ), while obtaining models more than 4 times smaller.
Moreover, The PFA ﬁne strategy largely outperforms the full model trained from scratch on the target domain ( Full scratch ).
The PFA-KL recipes when adapting from D C10 A are very similar to each other, independently of the target doamin (Fig. 3d), while those obtained adapting from D C100 A vary (Fig. 3b).
We believe that the fact that the target domain is a subset of D C100 A and not of D C10 A is the explanation to this effect.
In the
D C100 A case responses are stronger and selective, since the images used for PFA were also used for training, leading to stronger correlations.
PFA is able to exploit such effect, obtaining recipes adapted to the complexity of the task.
Note how PFA obtains recipes with more ﬁlters for the 10 class subsets (Fig. 3b).
On the other hand, the model trained on D C10 A has never seen the target domain, therefore this generates less correlation.
As a result PFA generates more uniform recipes, as shown in Fig.
3d.
These results show how PFA is able to transfer knowledge to different domains.
When the original network had been exposed to the target domain (Figs. 3a, 3b), PFA is able to exploit such knowledge and create recipes adapted to the task.
On the other hand, PFA obtains more uniform (and still accurate, Figs.
3c, 3d) recipes when the original network had not been exposed to the target domain.
C ONCLUSIONS
Two effective, and yet easy to implement techniques for the compression of neural networks using
Principal Filter Analysis were presented: PFA-En and PFA-KL.
These techniques exploit the inherent correlation of ﬁlter responses within layers to compress networks without compromising accuracy.
These techniques can be applied to the output response of any layer with no knowledge of the training procedure or the loss function.
While easy to implement, both algorithms surpass state of the art results in terms or compression ratio with the advantage that PFA-KL is parameter free and PFA-En has only a single intuitive parameter: the energy to be preserved in each layer or a desired network characteristic (such as a target footprint or FLOPs).
By performing spectral analysis on the responses rather than the weights, PFA allows users to take advantage of transfer learning in order to perform domain adaptation and derive smaller specialized networks that are optimally designed for a speciﬁc task, while starting from the same base model.
It is interesting to observe that all compression algorithms in the state of art, including PFA-KL, do not converge if applied repeatedly.
This is due to the retraining step which alters the weights in order to optimize an objective function that does not take into account compression criteria.
In practical applications this is not a limitation since most of the models tend to be over speciﬁed and users can apply compression algorithms until the accuracy falls below an acceptable level.
However, it is an interesting theoretical limitation that could inspire future work.
Speciﬁcally for PFA, this could be a strategy that can also suggest expanding layers so that an optimal size could be reached at convergence.
Appendices A S IMPLE CNN ARCHITECTURE
SimpleCNN is composed of the following layers: 3 conv layers of size 96x3x3, a drop-out layer, 3 conv layers of size 192x3x3, drop-out layer, 1 conv layer of size 192x3x3, 1 conv layer of size 192x1x1, 1 conv layer of size [number of classes]x1x and ﬁnally an average pooling before the softmax layer.
We use batchnorm and ReLU activations after every convolutional layer.
The drop-out probability is set to 0.5.
B R ES N ET -56 RESULTS ON CIFAR-10 AND CIFAR-
Results of PFA applied to Resnet-56 on CIFAR-10 and CIFARO-100 are shown in B.1a and B.1b
[Figure]