RASPA: CO - EXPLORING MODEL LOW -RA NKNESS AND SPA RSITY FOR COMPACT NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Low-rank decomposition and sparsiﬁcation are two important techniques for deep neural network (DNN) compression.
To date, these two popular yet distinct approaches are typically used in a separate way; while their efﬁcient integration for better compression performance is little explored.
In this paper we perform systematic co-exploration on the model low-rankness and sparsity towards compact neural networks.
We ﬁrst investigate and analyze several important design factors for the joint low-rank factorization and pruning, including operational sequence, low-rank format, and optimization objective.
Based on the observations and outcomes from our analysis, we then propose RASPA, a uniﬁed DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efﬁcient way.
Empirical experiments demonstrate very promising performance of our proposed solution.
Notably, on CIFAR-10 dataset, our approach can bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20,
ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet-40), respectively.
On
ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.
I NTRODUCTION
Deep neural network (DNN) has served as the backbone machine learning technique in many modern intelligent systems.
To facilitate the low-cost deployment of DNN on the resource-constrained platforms, model compression , as a powerful strategy that can efﬁciently reduce DNN model size, has been extensively studied in recent years.
To date, numerous compression approaches have been proposed to provide compact DNNs for many practical applications (Han et al.
(2015b)
; Gong et al.
; Liao et al.
Among various types of model compression techniques, sparsiﬁcation (a.k.a., pruning) and lowrank decomposition are two representative and popular solutions (Wang et al.
; Gao et al.
; Li et al.
(2021a)
; Xu et al.
As revealed by their names, the low-rank and sparse methods aim to explore and leverage the potential low-rankness and sparsity of the uncompressed
DNNs, respectively.
In practice, supported by the widely existed overparameterization phenomenon (Denil et al.
; Han et al.
(2015b))
, such hypothesized structure-level redundancy usually exists and thus it can be safely removed while still preserving high model performance.
Co-exploring Low-rankness & Sparsity: Motivation.
Considering the current prosperity of these two methods and their very distinct structural assumptions, an interesting and promising research topic is to explore the efﬁcient integration of low-rank and sparse approaches towards a better model compression solution.
As indicated and observed by Yu et al.
(2017), DNN models tend to exhibit both low-rankness and sparsity simultaneously.
For instance, the smooth components in the weight ﬁlters can be represented in the low-rank space, and meanwhile some other important information is sparsely scattered.
Evidently, fully leveraging such co-existence of these structure-level patterns, if being performed properly, can potentially bring a powerful compression solution with attractive performance.
Existing Works.
Unlike the current extensive research activities on individual low-rank and sparse methods, the investigations on integrating these two approaches, in an efﬁcient and non-trivial way, are little explored.
To date, only very few efforts study the joint exploration of low-rankness and sparsity for DNN model compression.
As the pioneering work along this direction, Yu et al.
(2017) develops a singular value decomposition (SVD)-free approach to closely approximate original DNN model via combining sparse representation and low-rank matrix factorization.
Built on the interesting connection between ﬁlter decomposition and ﬁlter pruning, Li et al.
(2020) interprets the decomposition and pruning of convolutional ﬁlter in a uniﬁed perspective.
Most recently, Li et al.
(2021b) proposes a collaborative compression scheme to integrate SVD into model sparsiﬁcation.
By adopting a multi-step heuristic removal strategy, this post-training approach achieves promising task and compression performance.
Unanswered Questions.
Although these prior works have demonstrated the huge potentials and attractive beneﬁts of jointly decomposing and pruning, the systematic investigation on their efﬁcient integration is still missing.
To be speciﬁc, several fundamental and critical questions, whose answers will directly impact the integration scheme and overall compression performance, have not been comprehensively explored yet.
For instance, because pruning and decomposition can be jointly performed in several different ways, such as in parallel (Yu et al.
or in sequence (Li et al.
(2021b)), which collaborative strategy is the best ﬁt for the target DNN compression task?
Also, considering low-rankness can be exploited from different perspectives, which type of low-rank approach should be adopted?
The matrix factorization used in Li et al.
(2020; 2021b)?
Or even highorder tensor decomposition?
In addition, to achieve promising compression performance, what is the most suitable optimization objective that the integration scheme should aim?
The approximation error focused in Yu et al.
The low-rankness/sparsity regularized loss in Yang et al.
Or some other new alternatives?
Technical Preview and Contributions.
To answer these questions and develop efﬁcient integrated model compression solution, in this paper we perform systematic co-exploration on the model lowrankness and sparsity towards compact neural networks.
To be speciﬁc, we ﬁrst review and analyze several important design factors for the joint low-rank decomposition and pruning.
Based on the observations and outcomes from our analysis, we then propose RASPA, a uniﬁed DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efﬁcient way.
Overall, the contributions of this paper are summarized as follows:
• We systematically investigate and analyze the critical design knobs when co-exploring model low-rankness and sparsity, including operational sequence, low-rank format, and optimization objective.
Based on our qualitative and quantitative analysis, we propose several recommended design options for efﬁcient joint low-rank decomposition and pruning.
• We develop a uniﬁed framework that formulates the integration of low-rank decomposition and pruning to an optimization problem with low-tensor-rank and sparse constraints.
We then derive a training-aware approach to solve this challenging non-convex high-order tensor-format problem, and thereby leading to efﬁcient exploration of rich low-rankness and sparsity in the model.
• We empirically evaluate our proposed co-exploration solution for various DNN models on different datasets, and the experimental results demonstrate its very promising performance.
Notably, on CIFAR-10 dataset, our solution can bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet40), respectively.
On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.
R ELATED W ORK
Sparsiﬁcation.
Sparsiﬁcation, also known as pruning, has been extensively studied for model compression (Han et al.
(2015a);
Wen et al.
; Gao et al.
; Guo et al.
; Rao et al.
In general sparsifying a DNN can be realized via two ways.
The ﬁrst one is to use a certain criterion, e.g., weight magnitude (Han et al.
(2015a)),
to directly remove some part of the model, and then perform ﬁne-tuning to recover the accuracy.
The second one is to add the sparsity-induced regularization during the training, such as (cid:96) 1 or group lasso term (Wen et al.
, to enforce the sparsity on the model.
In addition, in order to achieve good balance between accuracy and complexity reduction, Gao et al.
; Guo et al.
; Rao et al.
(2021) also proposes dynamic pruning.
In such scenario, the DNN sparsiﬁcation is essentially performed in an input-aware way – which part of model should be pruned is dynamically determined by each input data.
Low-Rank Decomposition.
Low-rank decomposition is another popular DNN compression approach.
Based on different interpretation of the neural network models, the low-rank method can be categorized to matrix decomposition and tensor decomposition .
Matrix decomposition views the 4-D weight tensor as the folded matrix, and hence it ﬂattens the 4-D objective to 2-D format and decomposes the reshaped matrix to the product of two small matrices (Tai et al.
; Li & Shi (2018)
; Xu et al.
On the other aspect, tensor decomposition directly factorizes the 4-D weight tensor to multiple small tensor cores without ﬂattening operations.
Such explicit high-order processing, by its nature, can better preserve the important spatial information and correlation existed in the weight tensors.
To date several tensor decomposition techniques, such as tensor train,
Tucker and tensor ring etc., have been used for DNN model compression (Kim et al.
; Novikov et al. (2015); Wang et al.
Joint Pruning and Decomposition.
As observed by Yu et al.
(2017), a well-trained DNN tends to exhibit both sparsity and low-rankness simultaneously.
Motivated by this observation, some prior efforts propose to co-explore these two complementary properties for model compression.
As the pioneering work, Yu et al.
(2017) decomposes the weight tensors of a pre-trained DNN model into independent low-rank and sparse parts and minimizes the reconstruction error.
Different from this parallel scheme, Dubey et al.
; Li et al.
(2021b) adopt a sequential compression strategy via performing matrix factorization on a pruned model.
In addition, Li et al.
(2020) proposes to use the sparse/low-rank regularization term, instead of reconstruction error, to enforce the desired structural patterns.
Also, notice that all of the existing works focus on using either SVD-based or SVD-free matrix decomposition to exploit the low-rankness of DNN model.
C O - EXPLORING L OW - RANKNESS AND S PARSITY : A NALYSIS
As outlined in Section 2, the integration of low-rank decomposition and pruning can be speciﬁed by several important factors, including operational sequence, low-rankness format and the overall optimization objective.
The existence of such large variety of different factors and their combinations, by its nature, calls for the systematic investigation on the best-suited co-exploration scheme for DNN compression.
Such analysis framework, if being properly developed, can facilitate the optimal selection of various design factors already proposed in the existing literatures.
More importantly, the outcome from this systematic study will further guide and provide the better integration choices that have not been discovered before.
Questions to be Answered.
Next we analyze the critical design knobs and factors for efﬁcient coexploration on model low-rankness and sparsity.
To that end, three important questions need to be answered.
Question #1: What is the more suitable operational sequence when jointly low-rank decomposing and pruning DNN models?
Analysis.
In general, the co-existence of model low-rankness and sparsity can be explored in different ways (see Figure 1).
For instance, as adopted in Yu et al.
(2017), a well-trained DNN can be closely approximated as the combination of a low-rank component and a sparse component.
In other words, the two types of structure-level properties are imposed and leveraged in a spatially parallel way , and we denote this strategy as L+S , where L and S represent low-rank decomposition and sparsiﬁcation, respectively.
On the other hand, the joint use of factorization and pruning can also be performed in a temporally sequential way .
As illustrated in Figure 1, the original model can be ﬁrst imposed with low-rankness (or sparsity), and the size of the resulting partially compressed model can be further reduced by the second-stage pruning (or low-rank decomposition).
Following the similar notation, such sequential operation can be denoted as S(L) and L(S) .
In practice L(S) is a preferable choice that has been adopted in the prior works (Dubey et al.
; Li et al.
(2021b)).
Our Proposal.
Among the above described three general operational schemes, we believe L+S is the more suitable choice when considering to integrate pruning and decomposition together for model compression.
This is because unlike S(L) and L(S) , which ultimately still produce the compressed model in a single representation (sparse or low-rank) space, L+S enables the simultaneous representation of rich information of DNN models across different subspace, and thereby better preserving the structural characteristics and reducing the potential information loss.
To verify our hypothesis, we examine the approximation error incurred by three integration schemes.
As shown in Figure 2, with the same compression ratio for the weight tensor of one layer of a pre-trained ResNeton CIFAR-10 dataset, L+S shows much lower approximation error than its counterparts, especially in high compression ratio region.
This experimental phenomenon demonstrates that L+S scheme indeed can capture both the low-rank and sparse characteristics of DNN model in an efﬁcient way.
Question #2: What is the best suitable lowrank decomposition approach used when coexploring low-rankness and sparsity?
Analysis.
From the perspective of linear algebra, the low-rankness of a DNN model can be exploited using different ways.
As illustrated in Figure 3, for an example convolutional layer, imposing the low-rank structure can be realized by performing simple matrix factorization or high-order tensor decomposition.
Speciﬁcally, Yu et al.
(2017) chooses
SVD-free method to factorize DNN model and obtain the low-rank component, and Li et al.
(2021b) proposes to use SVD-based decomposition to serve as the second-stage compression approach in its adopted L(S) scheme.
Notice that though the weights of convolutional layer essentially form a 4-D tensor format, the existing works exploit the low-rankness via using matrix decomposition – the 4-D tensor needs to be ﬁrst ﬂatten to a 2-D matrix and it is then factorized to two small matrix components.
Our Proposal.
We argue that the high-order tensor decomposition, the option that has not been explored in the integration scheme before, is the better choice than the low-order matrix decomposition adopted in the existing works.
This is because as a reshaping-free technique that can directly factorize the tensor-format data to multiple tensor cores, tensor decomposition, such as Tensor Train (TT) and Tucker, can naturally capture and preserve the important spatial information and correlation of the original weight tensors in a more efﬁcient way.
Therefore, less information loss is expected after performing low-rank tensor-based DNN compression.
To verify our hypothesis, we compare the feature maps of the compressed convolutional layer of ResNet-20 on CIFAR-10 dataset using dif- ferent low-rank methods.
As visualized in Figure 4, compared with the matrix decomposition-based approach with the same compression ratio, tensor decomposition can make the output feature map of the compressed layer much more similar to the feature map of the original uncompressed layer.
In other words, low-rank tensor method can provide better preservation of the important feature information and thus it can bring potential higher model compression performance.
Question #3: What is the best suitable optimization objective that the integrated compression scheme should aim?
Analysis.
To efﬁciently realize the joint exploration of model low-rankness and sparsity with promising compression performance, different optimization strategies have been proposed in the existing works.
For instance, Yu et al.
; Ma et al.
(2019) aim to minimize the difference between the original weight matrix/tensor and the approximated reconstruction.
In addition, Ma et al.
(2019) proposes to explicitly add the low-rank and sparse regularization terms to the overall objective function, which can guide the training-aware procedure to enforce the desired structural patterns.
Our Proposal.
Different from the existing approximation error-centered or regularized loss-based solutions, we propose that the efﬁcient co-exploration scheme should be interpreted as the optimization procedure with the low-rank and sparse constraints.
Our rationale lies on two important observations of the inherent drawbacks of the prior efforts.
First, the approximation strategy adopted in Yu et al.
; Ma et al.
(2019) focuses on making the reconstructed model approach the original model as close as possible.
However, since 1) the approximation error always exists; and 2) the original model is not the only choice to achieve the desired accuracy, such strategy inherently can only search the low-rank and sparse components in a limited exploration space, thereby affecting the overall compression performance.
Second, though adding the regularization terms into loss function indeed facilitates the extraction of low-rank and sparse patterns, the effect of such simple regularizing method is still limited, especially considering the efforts of pushing for sparsity and for low-rankness may interfere with each other, thereby potentially causing unexpected conﬂicts.
Instead, by explicitly imposing the low-rank and sparse constraints on the overall optimization problem, these two structural requirement can be simultaneously satisﬁed with the proper use of optimization technique (to be discussed in Section 4).
As reported in Appendix A, our proposed constrained optimization strategy can successfully impose the desired low-rankness and sparsity onto the DNN models efﬁciently.
Summary of Our Analysis.
(cid:182)
Performing joint low-rank decomposition and pruning in a spatially parallel way ( L+S ) is the preferable operational sequence.
(cid:183) High-order tensor decomposition is the most suitable choice for the low-rank approach used in the integrated compression scheme.
(cid:184)
Imposing low-rankness and sparsity as the direct hard constraints on the loss optimization should be adopted to better satisfy the desired structural requirement.
C O - EXPLORING L OW - RANKNESS AND S PARSITY : O UR M ETHOD
Based on the above three key takeaways obtained from Section 3, in this section we propose a uniﬁed framework to co-explore low-rankness and sparsity in an efﬁcient way.
To be speciﬁc, we ﬁrst formulate the integration of low-rank tensor decomposition and pruning to a uniﬁed optimization problem, and then develop an efﬁcient algorithm to solve this non-convex high-order tensor-format problem.
4. P ROBLEM F ORMULATION
Recall that the analysis in Section 3 brings three important observations/proposals: using L+S operational sequence, choosing tensor decomposition, and directly imposing hard constraints.
Built on such three fundamental principles, we are now ready to formulate the integration of pruning and tensor decomposition to a uniﬁed optimization problem.
To be speciﬁc, given an uncompressed DNN model with weight tensor W ∈ R
O × I × K × K of each layer, our goal is to ﬁnd another compact model with weight tensors
L + S , which consists of low-rank component
L ∈ R
O ×
I × K × K and
sparse component S ∈ R
O × I × K × K for each layer, to minimize the following loss function:
[Equation]
where f ( · ) is the loss over the entire training dataset, and γ 0 , γ 1 , · · · , γ d and κ are the desired tensor ranks and the number of non-zero entries for L and S , respectively.
Notice that without loss of generality, we choose tensor train (TT) decomposition as the component high-order lowrank method in our framework.
So here d is the number of decomposed tensor cores with TT decomposition.
4. O PTIMIZATION
Directly optimizing problem (1) is challenging because of the co-existence of the non-differentiable rank ( · ) and card ( · ) as well as its inherent high-order tensor format.
To efﬁciently solve this problem, we propose to leverage alternating direction optimization method to split these two constraints.
To be speciﬁc, after introducing two auxiliary variables (cid:98) L and (cid:98) S that represent the desired low-TTrankness and sparsity in the optimization process, problem (1) can be then rewritten as:
[Equation]
where P =
L | rank ( L ) ≤ γ 1 , · · · , γ d } is the set of all tensors that satisfy the low-tensor-rank constraint, and Q = { S | card ( S ) ≤ κ } is the set of all tensors that satisfy the sparse constraint.
Then, we further relax the hard constraints to the corresponding augmented Lagrangian form and now we only need to optimize the following new constraint-free min-max problem:
[Equation]
where U and V are the dual multipliers associated to L and S , respectively, and λ is the penalty parameters.
To solve this minmax problem, we can split it into three separated parts, and independently optimize them in an iterative way.
Update L and S with SGD.
The ﬁrst independent optimization objective can be formulated as:
[Equation]
Since there are no hard constraints on the target variables L and S , standard DNN optimizer (e.g., stochastic gradient descent (SGD)) can be directly applied with learning rate α as:
[Equation]
Update (cid:98) L with TT Decomposition.
To update the introduced (cid:98) L , the optimization objective is:
[Equation]
Because (cid:98) L is strictly constrained to stay in the low-tensor-rank set P , the desired update can be performed using an analytical solution via TT-rank truncation, i.e.
[Equation]
To realize such truncating operation, we ﬁrst deﬁne a temporary tensor T =
L + U and reshape it as a new tensor (cid:101) T ∈ R ( K × K )
×
( O 1 × I 1 ) ×···× ( O d × I d ) with O = (cid:81)
d k =
I
k
, I = (cid:81)
d k =
I k .
Then (cid:101) T can be decomposed to d + 1 TT-cores as:
[Equation]
where C 0 ∈ R K × K , C j ∈ R R j − 1 ×
I j ×
I j × R j , j = 1 , · · · , d .
In this TT-format, the dimensions of
TT-ranks in TT-cores are truncated to the desired target, i.e., C (cid:48)
j = C j (1 : γ j − 1 , : , : , 1 : γ j ) .
After that we use the truncated TT-cores to recover the original tensor via:
[Equation]
And ﬁnally (cid:101) T (cid:48) is reshaped to the original shape of (cid:98) L to serve as the updated (cid:98) L .
Update (cid:98) S with Projection.
For updating (cid:98) S , the third optimization objective is:
[Equation]
Similar to the low-tensor-rank (cid:98) L , the sparse-constrained (cid:98) S can also be analytically updated as (cid:98) S
[Equation]
proj
[Equation]
S
[Equation]
12) where proj ( · ) is the projection that removes the smallest values to ensure that the updated (cid:98) S can satisfy the sparse constraint.
Update Multipliers U , V .
Upon the update of (cid:98) L and (cid:98) S , the dual multipliers U and V can be then directly updated as:
[Equation]
Notice that after the iterative update ﬁnishes, the low-rank component L is explicitly decomposed to TT-cores { C } dj =0 , and the entire compressed model consisting of TT-cores and sparse part S is ﬁnally ﬁne-tuned with standard SGD.
The overall RASPA algorithm is summarized in Algorithm 1.
E XPERIMENTS 5. E XPERIMENTAL S ETTING
Dataset and Baseline.
We evaluate our proposed approach on two image classiﬁcation datasets (CIFAR-10 and ImageNet).
For experiments on CIFAR-10 dataset, three CNN models (ResNet-20,
ResNet-56 and DenseNet-40) are compressed and tested.
For experiments on ImageNet dataset, we evaluate our approach for ResNet-50 and compare its performance with state-of-the-art model compression methods.
Hyperparameter.
All the experiments are conducted using SGD optimizer with batch size, momentum and weight decay as 128, 0.
and 0.0005, respectively.
The learning rates adopted in the optimization and ﬁne-tuning process are set as 0.1 and 0.005, respectively.
Within the total 180 epochs, the learning rate is divided by 5 at epoch 54, 108, 144 andgradually.
The entire training procedure is performed on RTX 3090 GPUs with Pytorch 1.8.1.
5. E VALUATION AND C OMPARISON ON CIFAR-10 D ATASET
Table 1 shows the evaluation results of the compressed ResNet-20, ResNet-56 and DenseNet40 models on CIFAR-10 dataset.
For each model, we compare the performance of our proposed RASPA with several types of compression methods, including decompositiononly ( L ), pruning-only ( S ), ﬁrst-pruning-thendecomposition ( L(S) ), and layer-wise either-pruning-or-decomposition ( S/L) ).
ResNet-20.
For ResNet-20 model, the proposed RASPA solution can bring 1.25% accuracy increase over baseline model with 70.4% and 71.1% model size and FLOPs reductions, respectively.
With even more aggressive compression strategy aiming 85.3% smaller model size and 86.1% fewer
FLOPs, our approach can still enable 0.88% higher accuracy than the original ResNet-20 model.
ResNet-56.
For ResNet-56 model, our approach can bring 1.02% accuracy increase over baseline model with 37.5% and 39.3% model size and FLOPs reductions, respectively.
When we perform more aggressive compression with 65.6% and 66.0% fewer parameters and computations, our compressed ResNet-56 can still enjoy 0.64% higher accuracy than the original uncompressed model, thereby exhibiting very superior performance than other state-of-the-art DNN compression methods.
DenseNet-40.
For DenseNet-40 model, our proposed sparsity/low-rankness co-exploration can bring 0.16% accuracy increase over the baseline model with 52.4% and 61.3% model size and
FLOPs reductions, respectively.
Moreover, with further higher compression effect, our RASPA approach can still enable 0.07% higher accuracy than the original uncompressed model with 65.3% and 74.5% fewer parameters and computations, respectively; while the existing approaches suffer accuracy loss with even lower compression ratio.
5. E VALUATION AND C OMPARISON ON I MAGE N ET D ATASET
Table 2 summarizes the compression performance of our approach and other existing works for
ResNet-50 on ImageNet dataset.
It is seen that our RASPA solution can bring 0.52% accuracy increase over baseline model with 48.7% fewer parameters.
When targeting for generating more compact model, our approach can still achieve high performance – it only has 0.25% accuracy drop with 58.6% model size reduction, which means it shows better test accuracy than its counterparts with even higher compression ratio.
5. D ISCUSSION & A NALYSIS
To obtain deep understanding of our proposed approach, we also perform some empirical analysis and ablation studies on the co-exploration procedure.
The details are referred to Appendix A.
C ONCLUSION
In this paper we propose to systematically co-explore DNN low-rankness and sparsity for efﬁcient model compression.
By performing comprehensive analysis on critical design factors, we propose
[Table] [Table]
RASPA, a uniﬁed compression framework that can capture model low-rankness and sparsity simultaneously and efﬁciently.
Evaluation results show that our proposed approach can bring signiﬁcant model size and computational cost reductions while still preserving high model accuracy
[Table]
A D ETAILS OF E MPIRICAL A NALYSIS IN S ECTION 5.
In this section we present the details of our empirical analysis and discussion in Section 5.4.
Notice that the example model investigated and evaluated here is ResNet-20 on CIFAR-10 dataset.
Simultaneously Obtaining Low-rankness and Sparsity.
Figure 5 shows the loss curves during the model optimization procedure.
Notice that here besides overall training loss, the individual lowrank and sparse loss component, which directly reﬂects the progress of enforcing low-rankness and sparsity, respectively, is also explicitly reported in this ﬁgure.
It is seen that our proposed approach indeed successfully imposes the desired low-tensor-rankness and sparsity with hard constraints onto the model, and thereby ensuring that the compressed model can fully exhibit both low-rank and sparse characteristics after the optimization.
Effect of Optimization Procedure.
We also study the beneﬁts of using our proposed optimization procedure described in Algorithm 1 to solve problem (2).
Here we compare our approach with a direct method that performs TT decomposition and pruning on the uncompressed model straightforwardly with the same TT-ranks and sparsity settings.
In addition, the same ﬁne-tuning process that RASPA adopts is also applied in this direct method.
Figure 6 shows the comparison results with respect to different compression ratios.
It is seen that our proposed optimization procedure brings signiﬁcant accuracy increase.
Visualization.
Figure 7 visualizes the weight tensor of one convolutional layer in a pre-trained
ResNet-20 model before and after performing our proposed compression approach.
Here the visualization of the low-rank and sparse components of the compressed layer is also illustrated in this ﬁgure.
It is seen that most of the weight information is preserved in the low-rank component, and meanwhile the sparse component contains some spatial pattern as well.
B D ETAILS OF A NALYSIS IN S ECTIONB. M ORE RESULTS FOR ANALYSIS IN Q UESTION #
Table 3,4 and 5 show the layer-wise approximation errors incurred by three operational sequences ( L+S , S(L) and L(S) ).
Here the baseline models include well-trained ResNet-20, ResNet-56 on
CIFAR-10 dataset
[Table]
and ResNet-50 on ImageNet dataset
[Table]
and the compression ratio is set as 3.0 for all the layers.
It is seen that L+S scheme always brings the smallest approximation errors
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
B. M ORE RESULTS FOR ANALYSIS IN Q UESTION #
Table 6 shows the difference (in term of mean square error (MSE)) between the output feature maps of original layer and the compressed one in ResNet-20 on CIFAR-10 dataset.
Here SVD and tensor train (TT) are adopted for matrix decomposition and tensor decomposition, respectively.
It is seen that, with the same or even higher compression ratio, high-order tensor decomposition always brings smaller approximation error than the matrix decomposition..
Table 7 shows the difference between the output feature maps of one original layer and the compressed version in ResNet-20 on CIFAR-10 dataset and the difference of the ﬁnal accuracy.
Here
SVD, Tucker and TT are adopted for low-rank decompositions.
It is seen that high-order tensor decomposition always brings smaller approximation error than the matrix decomposition and
TT always has the least impact on the ﬁnal accuracy.
[Table]
.conv (32, 32, 3, 3) 0.
4.21 l2b7.conv (32, 32, 3, 3) 0.
3.51 l2b8.conv (32, 32, 3, 3) 0.
4.12 l2b8.conv (32, 32, 3, 3) 0.
3.53 l2b9.conv (32, 32, 3, 3) 0.
4.21 l2b9.conv (32, 32, 3, 3) 0.
l3b1.conv (64, 32, 3, 3) 0.
6.26 l3b1.conv (64, 64, 3, 3) 0.
7.99 l3b2.conv (64, 64, 3, 3) 0.
6.92 l3b2.conv (64, 64, 3, 3) 0.
7.17 l3b3.conv (64, 64, 3, 3) 0.
7.08 l3b3.conv (64, 64, 3, 3) 0.
6.62 l3b4.conv (64, 64, 3, 3) 0.
7.88 l3b4.conv (64, 64, 3, 3) 0.
6.58 l3b5.conv (64, 64, 3, 3) 0.
7.91 l3b5.conv (64, 64, 3, 3) 0. 7.91 l3b6.conv (64, 64, 3, 3) 0.
7.65 l3b6.conv (64, 64, 3, 3) 0.
5.08 l3b7.conv (64, 64, 3, 3) 0.
6.42 l3b7.conv (64, 64, 3, 3) 0.
3.95 l3b8.conv (64, 64, 3, 3) 0.
4.98 l3b8.conv (64,
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C D ISTRIBUTION OF TT-R ANK AND S PARSITY R ATIO
Table.
8, 9, 10 and 11 list the layer-wise TT-rank and sparsity ratio of four compressed model.
Here Model I and Model II in each table correspond to two reported models of each network with different Top-1 accuracy in Table 1 and 2.
[Table] [Table]
D O THER EXPERIMENTAL RESULTS
Please check Table 12, 13, 14 and 15 for other experimental results including different setting of compressed ResNet-20 and comparison with more related works from [L1] to [A3].
E R ELATED W ORK
Pruning.
Pruning is a very popular compression strategy that explores the sparsity of DNN models.
Existing pruning methods can be performed with different granularity, e.g., weight pruning [E1] and channel pruning [E2][E3], or with different pruning criteria, such as magnitude based [E1], rank based [E2], and nuclear norm based [E4].
Low-rank Decomposition.
Low-rank compression methods aim to explore the intrinsic linear correlations of weights.
In general, low-rank decomposition can be performed via exploring matrix decomposition [E5][E6] and tensor decomposition [E7].
For compressing convolutional neural networks, tensor decomposition is typically more suitable choice since high-order tensor decomposition can properly explore multidimensional correlation in the weight tensor; while the existing matrix decomposition-based works [E5][E6] suffer the loss of important spatial information incurred by inevitable tensor ﬂatten operation. [Table]
Co-exploring Sparsity/Low-rankness for Model Compression.
Recently, several papers have been proposed to co-explore sparsity and low-rankness for efﬁcient DNN model compression.
In particular, additive compression strategy (L+S) have been explored in some prior works [E8][E9][E10].
However, the low-rank component of all of these existing works are based on matrix decomposition.
As analyzed before, high-order tensor decomposition is a more suitable low-rank methods for CNN compression; while this important technique has not been explored by the existing additive compression efforts yet.
Also, [E8][E9] aims to closely approximate original models.
As we analyze in Question # 3, this strategy is not the optimal solution for DNN model compression, since our task goal is to generate a compressed model instead of close approximation to the original model.
Different from these existing works, our approach is built on tensor decomposition with constrained formulation as the optimization objective, thereby demonstrating better performance than [E8][E9][E10].
Co-exploring Sparsity/Low-rankness for Matrix Approximation/Estimation.
Co-exploring sparsity/low-rankness for efﬁcient matrix approximation/estimation has also been explored [E11E14].
However, the feasibility and efﬁciency of those general methods for DNN model compression is not veriﬁed and not clear yet.
To be speciﬁc, our approach aims to perform model compression on deep neural networks with high accuracy; while those general methods aim to approximate/estimate a matrix with low approximation error.
Consider modern DNN models consist of many large-size layers and they need maintain high accuracy on large-scale dataset, our approach is focusing a very different problem that these general methods do not work on.
Though it is possible that those general techniques may be applied to DNN model compression after certain unknown modiﬁcation, the feasibility is not really veriﬁed yet, especially on large-scale dataset such as ImageNet.
Also, the corresponding computational costs and accuracy performance are unknown.
ADMM-based Model Compression.
In recent years ADMM has been used in several model compression papers, including single pruning [E15][E16], single low-rank compression [E18], joint quantization and pruning [E17] and additive compressing [E8][E10].
Our approach focuses on additive compression with distinct difference from [E8][E10] and other ADMM-based works.
This is because low-rank approach used in [E8][E10] is based on matrix decomposition, which is not the best solution for compression CNNs; while we adopt high-order tensor decomposition in our proposed additive compression.
From the perspective of ADMM process, the projection towards tensor decomposition is much more complicated than the one for matrix factorization.
This is because 1) it is involved with advanced high-order tensor operation instead of straightforward 2-D matrix computation; and 2) tensor decomposition (e.g., TT) outputs multiple 4-D tensor cores; while ma- trix factorization only generates two 2-D matrices.
Such huge difference makes the corresponding projection on the decomposed tensor cores is signiﬁcantly different from the projection on low-rank used in [E8][E10] or sparse matrix used in [E15][E16][E17].
CVPR’
NeurIPS’
NeurIPS’
CVPR Wokrshop’
IJCAI’
CVPR’
ECML’
[Table]
Very Deep Convolutional Networks for Classiﬁcation and Detection ( [Table] TPAMI
[Table] GroupReduce Approximation Neural Language Model Shrinking,
NeurIPS
[Table]
Automated Multi-Stage Compression
of Neural
Networks
[Table]
Workshops 2019)
[Table]
Low-Rank Compression
of
Nets
the Rank of Each
Layer (CVPR 2020) [
L
Factorized Higher-Order CNNs with an Application to Spatio-
Temporal Emotion Estimation (
CVPR
Relevant pruning works: [
P1] “Learning-Compression”
Algorithms for Neural
Net
Pruning (CVPR 2018) [
[Table]
Automatic Neural Network Compression
Sparsity-Quantization Joint Learning A Constrained Optimization -based Approach (CVPR 2020) Relevant additive combinations work: [A1] More General and Effective Model Compression via an Additive Combination of Compressions (ECML 2021)
[A2] Handbook of Robust Low-Rank and Sparse Matrix Decomposition.
Applications in Image and Video Processing (CRC Publishers 2016) [A3] Compressing by Learning in a Low-Rank and Sparse Decomposition Form (IEEE Access 2019)
[Table]
l2b2.conv (32, 32, 3, 3) (1, 32, 20, 1) 0.
[Table]
l2b2.conv (32, 32, 3, 3) (1, 32, 20, 1) 0.
[Table]
conv (32,
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
l2b
[Table]
conv (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
conv (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
conv (
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.conv (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
l2b5.conv (
[Table] [Table]
[Table] [Table]
[Table]
l2b6.conv (32, 32, 3, 3) (1, 20, 20, 1) 0.
[Table] [Table]
l2b6.conv (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
l2b7.conv (32, 32, 3, 3) (1, 20,
[Table]
l2b7.conv (32, 32, 3, 3) (1, 16, 16, 1) 0.
l2b8.conv (32, 32, 3, 3) (1, 16, 16, 1) 0. l2b8.conv (32, 32, 3, 3) (1, 16, 16, 1) 0.
l2b9.conv (32, 32, 3, 3) (1, 16, 16, 1) 0. l2b9.conv (32, 32, 3, 3) (1, 16, 16, 1) 0.
l3b1.conv (64, 32, 3, 3) (1, 40, 30, 1) 0. l3b1.conv (64, 64, 3, 3) (1, 40, 30, 1) 0.
l3b2.conv (64, 64, 3, 3) (1, 40, 30, 1) 0. l3b2.conv (64, 64, 3, 3) (1, 40, 30, 1) 0.
l3b3.conv (64, 64, 3, 3) (1, 40, 30, 1) 0. l3b3.conv (64, 64, 3, 3) (1, 40, 30, 1) 0.
l3b4.conv (64, 64, 3, 3) (1, 40, 30, 1) 0. l3b4.conv (64, 64, 3, 3) (1, 40, 30, 1) 0.
l3b5.conv (64, 64, 3, 3) (1, 20, 20, 1) 0.
l3b5.conv (64, 64, 3, 3) (1, 30, 20, 1) 0.
l3b6.conv (64, 64, 3, 3) (1, 30, 20, 1) 0. l3b6.conv (64, 64, 3, 3) (1, 30, 20, 1) 0.
l3b7.conv (64, 64, 3, 3) (1, 30, 20, 1) 0. l3b7.conv (64, 64, 3, 3) (1, 30, 20, 1) 0.
l3b8.conv (64, 64, 3, 3) (1, 30, 20, 1) 0. l3b8.conv (64, 64, 3, 3) (1, 30, 20, 1) 0.
l3b9.conv (64, 64, 3, 3) (
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.conv (12, 276, 3, 3) (1, 20, 20, 1) 0.
[Table]
.conv (12, 288, 3, 3) (1, 20, 20, 1) 0.
[Table]
.conv (12,
[Table] [Table]
[Table]
conv (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
conv (
[Table] [Table]
[Table] [Table]
[Table]
conv (12, 336, 3, 3) (1, 30, 30, 1) 0.
[Table] [Table]
[Table]
b3l4.conv (12, 360, 3, 3) (1, 25, 25, 1) 0.
b3l5.conv (12, 372, 3, 3) (1, 25, 25, 1) 0.
[Table]
conv (12, 384, 3, 3) (1, 25, 25, 1) 0.
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
+1.
2.0 ×
[Table] [Table]
+2.
2.3 × 80% 91.
+1.
2.7 × 80% 91.
+1.
3.4 × 80% 91.
+1.
4.1 × 80% 91.
+1.
5.4 × 80% 91.
+1.
7.6 × 80%
[Table] [Table]