Accelerating Sparse Convolution for Efﬁcient Neural Network Inference
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Weight sparsity is a promising approach to reduce model size and computation costof convolutional neural networks (CNNs).
Nevertheless, non-zero weights oftendistribute randomly in sparse CNN models, introducing enormous difﬁculty to obtain actual speedup on common hardware (e.g., GPU) over their dense counterparts.
Existing acceleration solutions either require hardware modiﬁcations for irregularmemory access support, or reply on partially structured sparsity pattern.
Neither ofthese methods are capable of achieving fruitful speedup on convolution layers.
In this work, we propose an algorithm-software co-designed sparse convolution,based on a novel out-vector-wise (OVW) sparse pattern.
Building on the insightthat vertical vector integrity can preserve continuous memory access in IM2COL, OVW pattern treats a V × 1 vector as an entirety.
To reduce the error caused bysparsity, we propose an equivalent transformation process, i.e., clustering-basedchannel permutation, to gather similar rows together.
Experimental evaluationsdemonstrate that our method achieve an 1 . 7 × and 3 . 2 × speedup over the SOTAsolution and the dense convolution of ResNet50 on NVIDIA V100 at 75% sparsity,respectively, with only negligible accuracy loss.
More interestingly, comparing tothe SOTA solution that achieves speedups only on data with 60% sparsity or more,our method starts to obtain speedups on data with only 10% sparsity.
Introduction
Convolutional neural networks (CNNs) have yielded astonishing results in many important domainssuch as vision [8], language [19] and gaming [16].
With CNN algorithm develops rapidly, thestorage and computing overhead of CNN models grow exponentially.
To signiﬁcantly reduce both thecomputations and data accesses, weight sparsity has been taken as a promising approach to improvethe hardware efﬁciency.
Although succeeded in reducing computations and data accesses, the unconstrained, ﬁne-grainedsparsity fails to bring practical speedups on common GPUs.
Normally, sparsity is introduced in CNNs randomly, as synapses and neurons with negligible value will be skipped from computationand data accesses, for they have little affect to the following neurons. Such randomness causes a highoverhead in processing sparsity.
On NVIDIA V100, only with over 95% sparsity ratio[18, 3], thesparse matrix multiplication can achieve the same performance as the dense matrix multiplication.
Unfortunately, existing solutions either require hardware modiﬁcations, or only partially address theproblem by constrained with structured, coarse-grained sparsity, resulting high accuracy loss.
Theformal is to leverage the sparse matrix-matrix multiplication (e.g., SPMM) operation on GPU.
While directly applying SPMM on sparse CNNs can run even slower than dense CNNs [ ? ], productive
SPMM acceleration solutions[22, 14] often require their unique need for dedicated hardware supportto overcome discontinuous memory access, which is impractical. A software-only solution is moredesirable.
The latter is to leverage the general matrix-matrix multiplication (e.g., GEMM) operationon GPU.
Many efforts have been made to explore regular, structured sparsity to reduce the overheadof processing sparsity [4, 9, 5].
Block sparsity[4] manages to restore the spatial locality of matricesto a large extent, at the cost of a strict restriction on the non-zero structure of sparse matrices, whichsuffers from severe inference accuracy loss. Balanced sparsity[2, 20, 15], newly introduced on
NVIDIA A100 GPU[14], however, lacks of ﬂexibility in choosing model sparsity rate that only exact50% sparsity ratio could be deployed on this dedicated hardware.
These efforts achieve some palpableacceleration comparing to dense GEMM operation, but they all struggle to attain similar result onconvolution layers which has proven to be a greater challenge.
We ﬁnd that the key reason for the inefﬁciency is the introducing of IM2COL operation, which isrequired to convert GEMM to a convolution operation. The IM2COL operation transforms one inputmatrix into a larger one for multiplication with purely IO operations. As the matrix multiplicationprocess growing faster and faster, IM2COL operation begins to occupy an increasing proportion ofthe whole operation. However, modern DNN accelerators hides IM2COL’s IO latency with implicit
GEMM convolution algorithm. Thus GEMM-based sparse methods will be increasingly powerless inconvolution computation. In this paper, we present a novel implicit GEMM-based sparse convolution algorithm. Based onabove key ﬁnding, we propose a novel out-vector-wise (OVW) sparse pattern for this algorithm.
OVWsparse pattern is a carefully designed vector-wise sparsity pattern which can hold memory consistencyfor both input matrix while sharing a same amount of data reuse rate as the dense convolutionoperation by treating a V × 1 vector as an entirety.
We also propose an clustering technique assistingwith channel permutation to recover network accuracy of the OVW pattern.
These efforts assist ouralgorithm to establish a superiority acceleration results over other sparse kernels while maintainingthe high performance of CNNs. We practice our methodology on various CNN models, showing andominant acceleration rate over existing sparse convolution algorithm. What’s more, our GPU kernelstarts to demonstrate practical kernel speedup at an unprecedented low sparsity ratio comparing toits dense counterparts.
The acceleration possible model sparsity threshold of our implementation islowered to less than 10%, while prior arts can only reduce it to around 60%. This advancement inthreshold decline renders us an abundant ﬂexibility in choosing proportion of weights to be pruned ineach layer.
The main contributions of this paper are listed as follows:
• We propose a vector-based sparsity pattern,
i
e., the OVW pattern to balances inferenceaccuracy loss and computation efﬁciency using hardware-friendly means. • We implement a new GPU convolution kernel based on the OVW pattern.
The technique ofextracting ﬁlter location information further reduces inference runtime.•
We propose a heuristic clustering methods to obtain an appropriate channel permutationfor reducing accuracy drop during weight pruning.
This channel permutation operation isdetached from the inference procedure.• Our GPU kernel can accelerate convolution at a wide range of model sparsity rate.
With fewaccuracy loss, it can speed up ResNet50 by 1 .
7 × and 3 . 2 × over the SOTA solution and thedense cuDNN convolution on NVIDIA V100 GPU at 75% sparsity level.
Related work
2.1 Software-only Acceleration For Sparse CNN Model
Weight pruning has been a popular technique for efﬁcient CNN inference.
Early studies[7, 6]show that removing a large proportion amount of unimportant connections in CNN models does not necessarily lead to inference accuracy impairment.
The reduction in parameters helps exploitredundancy in CNN models, which requires less computations and data accesses.
However, for CNNinference, weight pruned sparse DNNs usually perform worse than dense CNNs, unless the CNNsparsity ratios are very large, i.e., very sparse CNNs.
Experiments[18, 3] report a threshold of sparsityratio at 95% for unstructured sparse CNN models to run faster than dense model on NVIDIA GPU.
In order to address this issue, methods other than unstructured sparsity are exploited.
Researchersexploit various constraints on sparsity pattern in exchange for computation efﬁciency.
A major domainof this region is ﬁlter pruning, where parameters of an entire ﬁlter are pruned or kept as a whole.
However, this direct modiﬁcation of channel size suffers sharp accuracy drop [10, 11, 17].
Moderatesparsity patterns are also examined, such as block sparsity[18], which is proposed to elevate spatiallocality of sparse matrix.
But this feature can achieve speedups only when sparsity ratios are largerthan 70%.
Tile-wise sparsity[5] renders weight patterns even more ﬂexibility.
Compared to previousmethods, balanced sparsity[15, 20] is more feasible with a recent support from NVIDIA A100 GPUwho directly optimizes 2:4 balanced sparsity.
Recent work Shfl _ BW [9] uses matrix transformationto utilize block sparsity’s computation efﬁciency while removing some of its constrains, furtherreducing the acceleration possible model sparsity threshold of sparsity ratio from 70% to 60%.
Different from prior works which can only work for very sparse matrices, in this paper, we propose asoftware-only sparse convolution for CNNs that can obtain speedup when the sparsity ratio is only10%.
2.2 GEMM Based Convolution
GEMM has been adopted widely to perform convolution.
Since most of the operations in the CNNlayers can be directly mapped to GEMM, all modern commercial-grade hardware accelerators such as
TPUs and GPUs provide a highly optimized general matrix-matrix multiplication (GEMM) operator. Thus GEMM-based convolution performs signiﬁcantly better than other convolution methods such as FFT, Winograd on these hardware.
This GEMM-based algorithm could be further divided into explicit matrix multiplication and implicitmatrix multiplication.
Explicit matrix multiplication prepares transformed inputs for the GEMMengine using the IM2COL operation.
IM2COL is an IO intensive operation, which brings in signiﬁcantworkload other than computation cost[1].
Implicit matrix multiplication merges these operationsfor more efﬁcient memory accesses.
It maintains and updates pointers of feature map in sharedmemory carefully while performing tile-based matrix multiplication simultaneously.
On NVIDIA V100 GPUs, explicit GEMM convolution is on average 120%, 126% and 142% slower than implicit
GEMM convolution on convolution layers of Alexnet, Resnet and Googlenet [21] .
Performing sparse convolution by GEMM is always through explicit matrix multiplication rather thanimplicit matrix multiplication.
The reason is because of the IM2COL operation, which is split intopieces in implicit matrix multiplication. This IM2COL division is extremely difﬁcult if not impossiblefor sparse matrix multiplication, as sparse matrices are compactly compressed and irregularly stored.
Therefore, GEMM-based sparse convolution is only implemented with explicit GEMM, which has tosuffer from the costly entire IM2COL operation. Yet, few studies have investigate the sparse convolution with implicit GEMM. As explicit GEMMperforms worse than implicit GEMM, sparse convolution with implicit GEMM has the potential toachieve higher performance and efﬁciency.
In this paper, we investigate the implicit GEMM basedsparse convolution to leverage the high performance GEMM on exist hardware.
Accelerating Sparse Convolution
In this section, we introduce our proposed sparse convolution, including the outer-vector-wise (OVM)pattern of sparsity for the proposed sparse convolution, and our detailed implementation on GPU.
3.1 OVW Pattern
Sparsity in matrix can be classiﬁed into three different patterns, as shown in Fig 1.
The ﬁrst sparsitypattern is element-wise(EW) pattern, corresponding to unstructured pruning, which evaluates eachparameter individually.
Having imposed no constraint on pruning, this pattern succeed at modelﬂexibility but struggled at actual deployment acceleration due to its irregular memory accesses.
Thesecond sparsity pattern is vector-wise pattern. Here we summarizes two types of vector-wise sparsitypattern, the inter-vector-wise(IVW) and the outer-vector-wise(OVW).
They both treat a V × 1 vectoras an entirety while the IVW pattern prunes certain proportion of weights inside each vector and the OVW pattern focuses on the entire vector of weights.
The third pattern is block-wise(BW) patternand it’s minimum pruning granularity is a V × V block.
This pattern has the highest computationefﬁciency but it’s inference accuracy loss is unsatisfactory.
In this work, we use the OVW pattern as it shares the advantages of VW pattern, which balancedcomputation efﬁciency of BW and network accuracy of EW.
The Shfl _ BW pattern is a variant ofthis pattern, who uses an extra channel reordering procedure to gather block-wise pattern utilities.
The OVW pattern, however, could be directly used in our convolution algorithm which denotes ahigher acceleration potential. Comparing to another vector-level sparse pattern the IVW pattern, alsoknown as the N:M structured sparsity, our approach holds no requirements for specialized hardwaresupports and it is much more elastic in selecting sparsity ratio of each layer.
Also the IVW patternstill faces the memory-bound issue, because the amount of redundant data that needs to be loadedinto shared memory each time is equal to its sparsity ratio.
3.2 Overview of the Proposed Sparse Convolution
Utilization of efﬁcient dense warp-level GEMM instruction is the key contributor of most of thesparse matrix acceleration methods.
As shown in Fig2, the OVW pattern sparse convolution canbe broken into multiple dense matrix multiplication of smaller size.
During a normal convolutionprocedure, inputs data is loaded to shared memory as block shape for block GEMM computation. In the loading process, a column of ﬁlter data loaded into shared memory shares a speciﬁc positionon the ﬁlter map where a continuous block on feature map is loaded to be computed with.
Severalcolumn fetched from ﬁlter together form the left input matrix of the block matrix multiplicationand their corresponding feature data blocks form the right input maatrix.
Noticing that this formingprocess of input matrix does not require the loaded ﬁlter columns to be continuous, meaning thatefﬁcient dense operations can also be performed by grouping some unrelated columns.
Buildingon this foundation, we could take in multiple columns of irrelevant column indices from the OVWpattern sparse matrix and handle it in the same way as the dense GEMM operation.
This resemblancebetween our convolution algorithm and the implicit GEMM convolution guarantees us similar overallcomputation efﬁciency.
This procedure only additionally introduces the loading instruction workloads of column indices dataand the calculation of the corresponding ﬁlter location compared to the original dense convolution.
Both of these workloads are negligible in comparison to the high computation cost reduced by sparsity. Vector length used in this format depends on the tile size that hardware shared memory could hold.
3.3 GPU Sparse Kernel
As shown in Algorithm 1, the convolution kernel contains two steps.
The ﬁrst step is to loaddata from both input matrix to shared memory.
Some threads use function load_column to loadcolumn vector of length TM from ﬁlter to shared memory with DY threads.
Threads then usefunction load_row to load row vector from feature map in the same way.
The second step callswarp matrix multiplication operands.
When calculating matrix A M × K × B K × N = C M × N , athree dimension parallelism ( DX, DY, DZ ) thread is employed[].
( DX, DY, DZ ) threads eachloops along ( M, N, K ) individually.
Several threads together use function Warp_MMA to multiplythe loaded matrix and writes back after results accumulation.
Each thread computes a tile matrixmultiplication of the size ( TM, TN, TK ) .
The procedure of calculating the exact pointer offset of feature map for corresponding ﬁlter columnscontains two steps.
Because during a convolution computation, corresponding location of a ij × b jkis not obvious.
After loading a ij into shared memory, ﬁrstly, GPU kernel has to fetch the columnindices of a ij in the original ﬁlter.
The column indices is then used to recover the exact position ofthis column in the ﬁlter map.
Secondly, the location offset of the corresponding data in feature map iscalculated and after which b jk can be ﬁnally located in the feature map.
3.4 Preprocessing Filter Location Information
If we only stores the column indices of sparse matrix, its location information has to be recoveredeach time before the corresponding activation is loaded.
Resembling to the location offset in featuremap, the corresponding data location in ﬁlter map is a constant for each thread during the wholeprocess.
If we prepare this information in advance and store its actual location instead of its columnindices, the location calculation process could be saved, thus introducing could be further accelerated. Extra storage occupation of this technique is two extra dimension indices data array of ﬁlter map,which takes merely 3% total storage of a compressed model with vector length=64, and 6% withvector length=32, in exchange of 10% run time reduction of Resnet50’s convolution layers in average. Considering the sparse model is already highly compressed, this additional model redundancy istotally acceptable. Data: row _ idx [] , filter [] , input []
Result: output [] 1 Shared memory
A [ TM ][ TK ], B [ TK ][ TN ], C [ TM ][ TN ]; 2 for Thread idx=1 to DX, idy=1 to DY, idz=1 to DZ do offset =
Cal_Thread_Offset( row _ idx [] , idx , idy , idz ); if idx < TK then
Load_column( A , filter [ idz ][ idx ] , TM , idy ); end if idy < TK then Load_row( B , input [ offset ] , TN , idx ); end
Syncthreads(); Warp_MMA( A , B , idx , idy );
Accumulate_Results( C ); Store( output , C ); 14 end 15 Return output;
Pruning Algorithm
[Figure]
4.1 Channel Permutation
[Figure] Our pruning methods can be divided into two steps: shufﬂing the ﬁlter matrix rows in each layer to adesirable one, applying an vector-wise pruning to the intermediate results.
Here we will explain why ﬁlter permutation will do no harm to the network inference.
Matrix multiplication only swaps the order in the output dimension and does not change the actual computation.
Permuted operation results can be recovered through a reversed permutation of the operand output.
As we only permutes the output channel of each layer, new order of the current layer will absorbedby the input channel dimension of the next GEMM-based layer(convolution or linear).
Fig. 3 showsone iteration for channel permutation between layer k and layer k +1 .
As we were reordering the output channel of layer k , the activation k is changed to the same order, but after we permutes theinput channel of layer k +1 , activation k + 1 is restored.
More weights value could be saved afterpermutation.
The same operation is then repeated on layer k [-+
-]+ 1 and so on, until every GEMM layerin the network is permuted.
This permutation transfer procedure allow us to choose a appropriatepermutation row order for every ﬁlter without altering the output of the network.
Rest of the layers such as the pooling layer and the activation layer involve no modiﬁcation alongchannel dimension thus are not affected by this process.
BN layers and the bias adding at the endof convolution layers and linear layers do not produce any new permutation order, but they have topermute according to the permutation passing through.
4.2 Row Clustering
Algorithm 2: Data: The original weight W , number of clusters k , number of selected column m . Result:
The reordered weight RW . 1 RW = empty ; 2 while W is not empty do Sort the columns of W by column variance; Build SampleW by selecting columns with the top- m largest variances;
Get the k clustered groups G = Balanced kmeans SampleW , k ); Select the group g with maximum sum;
Append g to RW ;
Remove g from W ; 9 end 10 Return RW ; The only remaining question is how to decide an appropriate permutation order. A heuristic indicatorevaluating the quality of a permutation is the sum of absolute weight value pruned, assuming that themethod that the preservation of more important weights corresponds to less inference accuracy loss. An obvious route is that the weight rows with shorter distance assigned to the same group as muchas possible.
Shfl _
BW
chooses the kmeans method for clustering, but after carefully experimentsthis idea, we found that the kmeans method does not suit this issue well.
For starters, the numberof elements in each group is set to be a ﬁxed value (vector length), and kmeans requires additionaloperations to meet this demand.
Also the data dimension(input channel multiplies ﬁlter height andwidth) is too large, while the amount of data and groups are relatively small.
Kmeans falls in localminima easily and the output cluster is extremely unstable.
We modiﬁed the kmeans to better adaptthe OVW pattern, which is able to palliate both symptom mentioned above.
Algo2 shows the key steps of our algorithm.
First, we construct a characteristic matrix by assemblingrows with highest variance, then cluster rows of this matrix to alleviate excessive dimension.
Weutilize a balanced clustering method[13] to solves the assignment of given cluster sizes.
During theclustering process, this problem is equivalent to a linear sum assignment problem.
We employ KuhnMunkres algorithm to solve the distance matrix and optimize the mean square error of it.
Secondly,for each clustering result, we only adopt the most important group under this feature to increasesoperation stability.
This group is removed from the original matrix and then the feature matrix isreconstructed and clustered.
Repeat the above steps until all data grouping completes, the permutedmatrix and its permuting order is obtained.
Evaluation5.1 Model Accuracy
We evaluate our method on several popular CNN models on NVIDIA V100 GPU.
We only calculatethe speedup of the convolution layers in the following results.
Table1 shows the accuracy of our
[Table]
method comparing to unstructured sparsity.
OVW permuted shows a better accuracy over OVWnon-permuted on all this network.
All the results in this table use the same ﬁne-tuning process.
We ﬁne-tune each network for 40 epochsafter pruning with the same learning rate of 0.0008.
Also, each layer can hold different sparsity ratiocredit to our acceleration for convolution at low sparsity ratio.
Vector length for OVW pattern here isset to the tile shape of highest speedup ratio. For example, the ﬁrst convolution layer of SqueezeNethas only 96 output channel. If vector length is 64, the second tile The layer will be only half loaded. In this case vector length is set to 32 to maximize computation resources utilization.
In depthwiseconvolution of MobileNet, because ﬁlter column weight shares no common feature data to computewith, vector length larger than 1 has no speed advantages.
5.2 Convolution Kernel Speedup
As shown in Fig4, we evaluate the speedup of our method on three popular CNN models.
We usecuDNN convolution operator as the dense baseline.
The ﬁrst three graph in Fig4 represent threetypical convolution shape in CNN models: small channel size with large feature map, mediumchannel size with medium feature map and large channel size with small feature map.
Our kernelcan accelerate all these types of convolution layer while exceeds at the twelfth convolution layer of
[Table] Resnet50 which is the most used kind of convolution layer 4 .
8 × and 3 .
87 × over cuDNN on V100 at80% sparsity, vector length 64 and 32.
5.3 Comparing Sparsity Patterns
We identify two vector-level sparsity pattern, balanced sparsity(NVIDIA 2:4)[15] and Shfl _ BW [9]for comparison.
Other vector-level sparsity pattern such as Tile-wise[5] are slower than the formertwo patterns.
Also these pattern lack implementation for convolution are.
Table2 contains results of Resnet50 on ImageNet directly copied from Shfl _ BW paper, where anexpensive method Grow and Prune[12] is used to recover its accuracy.
Grow and Prune is a sparsitypattern independent method.
Training from scratch for 990 epochs, this method can achieve highaccuracy, but it is sparsity method independent, meaning that any sparsity pattern can utilize it to getbetter accuracy.
We ﬁne-tune our pretrained Resnet50 model for 20 epochs with the learning rate0.001.
We lower the sparsity of our network to 70% sparsity where our method demonstrate 73.35%top1 accuracy and 2 . 79 × .
Exhibiting a better speed-accuracy trade of comparing to Shfl _ BWpattern.
The OVW pattern also achieves better speedup rate comparing to balanced sparsity whilerecovering full accuracy of the origin model.
To ensure fair comparison, we reproduce these results under the same setting on Resnet50 Cifar100,as shown in Fig5.
We ﬁne-tune each network for 40 epochs after pruning from pretrained densemodels with the same learning rate of 0.0008.
The OVW pattern dominates the speed-accuracytrade-off in vector-level sparsity.
Conclusion
Accelerating sparse convolution poses a greater challenge than accelerating sparse matrix multiplication.
In this work, we recognize a novel sparsity pattern, the OVW pattern which leverages modelaccuracy and inference acceleration.
Our GPU implementation outperforms all sparse accelerationapproaches exceedingly with a sparse pattern of similar ﬂexibility.
We hope this paper can ﬁll thevacancy of specialized sparse convolution kernel domain and our methodology can stimulate furtherexploitation on this sparsity pattern. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (
) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation
[N/A]
[Table]