Accelerating Sparse Convolution with Column Vector-Wise Sparsity
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Weight sparsity is a promising approach to reducing the model size and computationcost of convolutional neural networks (CNNs).
Nevertheless, non-zero weightsoften distribute randomly in sparse CNN models, introducing enormous difﬁcultyin obtaining actual speedup on common hardware (e.g., GPU) over their densecounterparts.
Existing acceleration solutions either require hardware modiﬁcationsfor irregular memory access support or rely on a partially structured sparsity pattern.
Neither of these methods is capable of achieving fruitful speedup on convolutionlayers.
In this work, we propose an algorithm-software co-designed sparse convolutionbased on a novel out-vector-wise (OVW) sparse pattern.
Building on the insightthat vertical vector integrity can preserve continuous memory access in IM2COL,the OVW pattern treats a V × 1 vector as an entirety.
To reduce the error causedby sparsity, we propose an equivalent transformation process, i.e., clustering-basedchannel permutation, to gather similar rows together.
Experimental evaluationsdemonstrate that our method achieves a 1 . 7 × and 3 . 2 × speedup over the SOTAsolution and the dense convolution of ResNet50 on NVIDIA V100 at 75% sparsity,respectively, with only negligible accuracy loss.
Moreover, compared to the SOTAsolution that achieves speedups only on data with 60% sparsity or more, our methodbegins to obtain speedups on data with only 10% sparsity.
Introduction
Recently, convolutional neural networks (CNNs) have yielded astonishing results in many importantdomains such as vision [8] and language [18].
With CNN algorithm developing rapidly, CNN models’storage and computing overhead grow exponentially.
To signiﬁcantly reduce both the computationsand data accesses, weight sparsity has been taken as a promising approach to improve hardwareefﬁciency.
Despite the success in reducing computations and data access the unconstrained, ﬁne-grained sparsityfails to bring practical speedups on common GPUs.
This is because unstructured sparsity generallyinduces tremendous access conﬂicts and load unbalances, which lowers GPU’s performance.
Forexample, on NVIDIA V100, under the sizes, the sparse matrix multiplication performs not faster thanthe dense matrix multiplication until the sparsity ratio is over 95% [17, 3].
Unfortunately, existing solutions either require hardware modiﬁcations or only partially address theproblem by being constrained with structured, coarse-grained sparsity, resulting in high accuracyloss.
The formal is to leverage the sparse matrix-matrix multiplication ( .g. ) operation on GPU.
While directly applying SPMM on sparse CNNs can run even slower than dense CNNs[17],productive
SPMM acceleration solutions[21, 14] often require their unique need for dedicatedhardware support to overcome discontinuous memory access, which is impractical.
The latter is toleverage the general matrix-matrix multiplication (e.g., GEMM) operation on GPU.
Recent works focus on structured sparsity with different sparse patterns to gain speedup beneﬁts fromweight sparsity.
Block sparsity[4] manages to restore the spatial locality of matrices to a large extent,at the cost of a strict restriction on the non-zero Balanced sparsity[2, 19, 15], newly introduced on
NVIDIA A100 GPU[14], however, lacks ﬂexibility in choosing model sparsity rate that only exact50% sparsity ratio could be deployed on this dedicated hardware.
These efforts achieve some palpableacceleration compared to dense GEMM operation, but they all struggle to attain similar results onconvolution layers which have proven to be a greater challenge.
To tackle these problems, here we present a novel sparse convolution acceleration algorithm featuredwith column-wise sparsity and implicit matrix multiplication.
Speciﬁcally, the proposed column-wisesparsity is dubbed the out-vector-wise (OVW) sparse pattern since the pattern sparsiﬁes a matrix bytreating a V×1 vector as an entirety, as shown in Figure 1.
During convolution, the OVW pattern canhold both strong memory consistency and high data reuse rates of input matrices using implicit matrixmultiplication.
Moreover, we propose to employ channel permutation and row clustering to improvethe accuracy of OVW sparse pattern-based CNNs.
Besides, a GPU kernel is carefully designed toensure that our OVW sparse pattern is supported by common GPUs. With these efforts, our algorithmpredominantly outperforms other sparse convolution acceleration algorithms on various CNN models.
More importantly, our algorithm can achieve the acceleration of convolutions even with a very lowweight sparsity ratio, e.g., 10%. Instead, prior arts can only work ﬁne when the weight sparsity ratiois over 60%.
The main contributions of this paper are listed as follows:
• We propose a vector-based sparsity pattern,
i
e., the OVW pattern to balance inferenceaccuracy loss and computation efﬁciency in a hardware-friendly manner.• We implement a new GPU convolution kernel to support the OVW pattern.
The kernelutilizes the technique of extracting ﬁlter location information which can further reduceinference runtime.•
We propose a heuristic clustering method to obtain an appropriate channel permutationfor reducing accuracy drop during weight pruning.
This channel permutation operation isconducted ofﬂine, which does not affect inference time.• Our GPU kernel can accelerate convolution at a wide range of model sparsity rates.
With fewaccuracy loss, the kernel can speed up ResNet50 by 1 .
7 × and 3 . 2 × , respectively over theSOTA solution and the dense cuDNN convolution on NVIDIA V100 GPU at 75% sparsitylevel.
Related work2.1 Software-only Acceleration For Sparse CNN Model
Weight pruning has been a popular technique for efﬁcient CNN inference.
Early studies[7, 6] showthat removing a large proportion of unimportant connections in CNN models does not necessarilylead to inference accuracy impairment.
Reducing parameters helps exploit redundancy in CNNmodels, which requires fewer computations and data accesses.
However, for CNN inference, weightpruned sparse CNNs usually perform worse than dense counterparts, unless the CNN sparsity ratiosare substantial, i.e., very sparse CNNs.
To address this issue, methods other than unstructured sparsity are exploited.
Researchers exploitvarious constraints on sparsity patterns in exchange for computation efﬁciency.
A primary domain ofthis region is ﬁlter pruning, where parameters of an entire ﬁlter are pruned or kept as a whole.
However, this direct modiﬁcation of channel size suffers a sharp accuracy drop [10, 11, 16].
Moderate sparsitypatterns are also examined, such as block sparsity[17], which is proposed to elevate the spatial localityof sparse matrices.
But this feature can achieve speedup only when sparsity ratios are larger than 70%.
Tile-wise sparsity[5] endows weight patterns with more ﬂexibility.
Compared to previous methods,balanced sparsity[15, 19] is more feasible with recent support from NVIDIA A100 GPU whichdirectly optimizes 2:4 balanced sparsity.
Recent work Shfl _ BW [9] uses matrix transformation toutilize block sparsity’s computation efﬁciency while removing some of its constraints. In this way,the threshold of weight sparsity ratio that enables acceleration is reduced from 70% to 60%.
Differentfrom prior works which only work for very sparse matrices, in this paper, our algorithm can achievespeedup when the sparsity ratio is only 10%.
2.2 GEMM Based Convolution
GEMM has been adopted widely to perform convolution and it performs signiﬁcantly better thanother convolution methods such as FFT, and Winograd on modern commercial hardware acceleratorssuch as TPUs and GPUs.
The GEMM-based algorithms could be further divided into two types:explicit matrix multiplication and implicit matrix multiplication.
Explicit matrix multiplicationuses IM2COL to adapt inputs for GEMM.
IM2COL is an IO-intensive operation, which brings insigniﬁcant workload other than computation cost[1].
Implicit matrix multiplication merges theseoperations for more efﬁcient memory accesses.
It updates pointers of feature map in shared memoryand performs tile-based matrix multiplication simultaneously.
On NVIDIA V100 GPUs, explicit GEMM convolutions consume on average 120%, 126%, and 142% in time compared to implicit
GEMM-based convolution on convolution layers of Alexnet, Resnet and Googlenet [20].
Yet, few studies have investigated the sparse convolution with implicit GEMM.
Performing sparseconvolution by GEMM is always through explicit matrix multiplication rather than implicit matrixmultiplication.
This is because of the IM2COL operation which is extremely difﬁcult if not impossiblefor sparse matrix multiplication, as sparse matrices are compactly compressed and irregularly stored.
As a result, implicit GEMM who does not have to suffer from the costly IM2COL operation has thepotential to achieve higher efﬁciency for sprase convolution.
In this paper, we investigate the implicit GEMM-based sparse convolution to leverage the high-performance GEMMs on existing hardware.
Accelerating Sparse Convolution
In this section, we introduce our proposed sparse convolution algorithm, including the OVW patternof sparsity for the proposed sparse convolution, its advantage in convolution computation and ourdetailed implementation on GPU.
3.1 The OVW Pattern
The OVW pattern belongs to the vector-wise(VW) pattern which is one of the three different patterncategories of sparsity in matrix.
As shown in Fig 1, the ﬁrst sparsity pattern is the element-wise(EW) pattern, corresponding to unstructured pruning, which evaluates each parameter individually.
Havingimposed no constraint on pruning, this pattern succeeded at model ﬂexibility but struggled at actualacceleration due to its irregular memory accesses.
The second sparsity pattern is the VW pattern,which can be further divided into the inter-vector-wise(IVW) and the OVW pattern.
They both treata V × 1 vector as an entirety while the IVW pattern prunes a certain proportion of weights insideeach vector and the OVW pattern focuses on the entire vector of weights.
The third pattern is theblock-wise(BW) pattern, and its minimum pruning granularity is a V × V block.
This pattern hasthe highest computation efﬁciency, but its inference accuracy loss is high as well.
In this work, weuse the OVW pattern since it shares the advantages of the VW pattern, which balances computationefﬁciency of BW and network accuracy of EW.
The Shfl _ BW pattern is actually a variant of thispattern, who uses an extra channel reordering procedure to gather block-wise pattern utilities.
3.2 The OVW Pattern’s Advantage in Convolution Computation
The biggest advantage of the OVW pattern is that it ﬁts the way an efﬁcient dense warp-level GEMMinstruction fetches input data.
This instruction is the key contributor to most of the sparse matrixacceleration methods. The reasons are as followed.
As shown in Fig 2, the OVW pattern-based sparse convolution can be broken down into multiple densematrix multiplications of smaller sizes.
During the loading process of a dense convolution procedure,a column of ﬁlter data loaded into shared memory shares a speciﬁc position on the ﬁlter map. Inthe meantime, a continuous block in the feature map is loaded accordingly to prepare convolution.
Several columns fetched from the ﬁlters together form the left input matrix of the block matrixmultiplication and their corresponding feature data blocks form the right input matrix.
Noticing thatthis forming process of input matrix does not require the loaded ﬁlter columns to be continuous,meaning that efﬁcient dense operations can also be performed by grouping some unrelated columns.
Based on this observation, we could take in multiple columns of irrelevant column indices from the OVW pattern sparse matrix and handle them in the same way as the dense GEMM operation.
Thissimilarity between our convolution algorithm and the implicit GEMM convolution guarantees ussimilar overall computation efﬁciency.
What’s more, other sparsity such as the Shfl _ BW pattern who is actually a variant of this pattern,uses an extra channel reordering procedure to gather block-wise pattern utilities. The OVW pattern,however, could be directly used in our convolution algorithm which denotes a higher accelerationpotential. Compared to N:M sparsity,one of the IVW pattern, our approach does not need specializedhardware supports and it is much more elastic in selecting the sparsity ratio of each layer.
Besides,the IVW pattern still faces the memory-bound issue, because the amount of redundant data that needsto be loaded into shared memory each time is equal to its sparsity ratio.
3.3 GPU Sparse Kernel Implementation
As shown in Algorithm 1, our convolution kernel implementation contains three steps.
The ﬁrst stepis to get the corresponding feature pointer offsets through original ﬁlter structure information recovery.
These parts of calculation are done by the function Cal _ Thread _
Offset .
The second step is toload data from both input matrices into shared memory.
Some threads use the function load _ columnto load a column vector of length TM from ﬁlters to shared memory with DY threads.
Threads thenuse the function load _ row to load a row vector from the feature map in the same way.
The third stepcalls warp matrix multiplication operators.
When calculating matrix A M × K × B K × N = C M × N, a three dimension parallelism ( DX, DY, DZ ) thread is employed.
( DX, DY, DZ ) threads loopalong ( M, N, K ) individually.
Several threads together use the function Warp _ MMA to multiplythe loaded matrices and write back after results accumulation is ﬁnished.
Each thread computes a tilematrix multiplication of the size ( TM, TN, TK ) .
The procedure of calculating the exact pointer offset of the feature map for corresponding ﬁltercolumns contains two steps.
During a convolution computation, the corresponding location ofa ij × b jk is not obvious.
Hence after loading a ij into shared memory, ﬁrstly, the GPU kernel has to fetch the column indices of a ij in the original ﬁlter.
The column indices are then used to recover theexact position of this column in the ﬁlter map.
Subsequently, the location offset of the correspondingdata in the feature map is calculated, after which b jk can be ﬁnally located in the feature map.
Data: row _ idx [] , filter [] , input []
Result: output [] 1 Shared memory
A [ TM ][ TK ], B [ TK ][ TN ], C [ TM ][ TN ]; 2 for Thread idx DX, idy DY, idz= DZ do
end end 14 end If only the column indices of sparse matrices are stored, their location information has to be recoveredeach time before the corresponding activation is loaded.
Like the location offset in the feature map,the corresponding data location in the ﬁlter map could be prepared in advance.
Because it is a constantfor each thread during the whole process.
Extra storage occupation of this technique is two extradimension indices data array of ﬁlter map, which takes merely 3% total storage of a compressedmodel with vector length=64, and 6% with vector length=32, in exchange for 10% run time reductionof Resnet50’s convolution layers on average. Considering that a sparse model is already highlycompressed, this additional model redundancy is totally acceptable.
Pruning Algorithm
[Figure]
In this section, we introduce our pruning algorithm for the OVW pattern, including the channelpermutation technique and our method to acquire a desired permutation order for it.
4.1 Channel Permutation
Our pruning method can be divided into two steps: shufﬂing the ﬁlter matrix rows in each layer andapplying vector-wise pruning.
Here we will explain why ﬁlter permutation will do no harm to the network inference.
Matrix multiplication only swaps the order in the output dimension and does not change the actual computation.
Permuted operation results can be recovered through a reversed permutation of the operand output.
As we only permute the output channel of each layer, the permuted order of the current layer will beabsorbed by the input channel dimension of the next GEMM-based layer(convolution or linear).
Fig 3shows one iteration for channel permutation between layer k and layer k +1 .
As we have reordered theoutput channel of layer k , the activation k is changed to the same order, but after we permute the inputchannel of layer k +1 , activation k +1 is restored.
More weights value could be saved after permutation.
The same operation is then repeated on layer k {++
+} and so on until every GEMM layer in the network ispermuted.
This permutation transfer procedure allows us to choose an appropriate permutation roworder for every ﬁlter without altering the output of the network.
The rest of the layers such as the pooling layers and the activation layers involve no modiﬁcationalong channel dimension thus are not affected by this process.
The BN layers and the bias added atthe end of convolution layers and linear layers do not produce any new permutation order, but theyhave to permute according to the permutation passing through.
4.2 Row Clustering
We use a row clustering method to obtain an appropriate permutation order.
A heuristic indicatorevaluating the quality of a permutation is the sum of absolute weight value being pruned, assumingthat the method that the preservation of more important weights corresponds to less inference accuracyloss.
An obvious route is that the weight rows with shorter distances are assigned to the same group as
The reordered weight RW .
much as possible.
Shfl _
BW
chooses the kmeans method for clustering, but after careful experiment,we found that the kmeans method does not suit this issue well.
For starters, the number of elementsin each group is set to be a ﬁxed value (vector length), and kmeans requires additional operations tomeet this demand.
Also, the data dimension(input channel multiplies ﬁlter height and width) is toolarge, while the amount of data and groups is relatively small.
Kmeans falls in local minima easilyand the output cluster is extremely unstable.
We introduced the balanced kmeans[13] to solve it andmodiﬁed it to palliate both symptoms mentioned above.
Algorithm 2 shows the key steps of our algorithm.
First, we construct a characteristic matrix byassembling rows with the highest variance, then cluster rows of this matrix to alleviate excessivedimension.
We utilize balanced kmeans to get equal size clusters.
In each iteration of balancedkmeans, instead of assigning each data vector to its nearest cluster center as the origin kmeansalgorithm, a distance matrix between all the vectors and the current cluster centers is formed. Weminimize the sum of distance while each cluster contains the same amount of data vector. Thisminimization problem can be converted to bipartite matching
and we employ the Kuhn-Munkresalgorithm to solve it.
Secondly, for each clustering result, we only adopt the most important groupunder this feature to increase operation stability.
This group is removed from the original matrix andthen the feature matrix is reconstructed and clustered.
Repeat the above steps until all data groupingcompletes, the permuted matrix and its permuting order is obtained.
Evaluation
5.1 Model Accuracy
We evaluate our method on several popular CNN models on NVIDIA V100 GPU.
We only calculatethe speedup of the convolution layers in the following results.
Table 1 shows the accuracy of ourmethod compared to unstructured sparsity where V stands for vector length.
“OVW permuted” showsa better accuracy over “OVW non-permuted” on all CNN models.
The upper bound of V in our kernel implementation is 64 and we tend to make it as large as possibleto maximize shared memory usage.
However, group convolution has no ﬁlter data reusage for vectorlength larger than its group size.
Similarly, V is set to 1 in depthwise convolution layers.
Besides that,the ﬁrst convolution layer of SqueezeNet has only 96 output channels.
If the vector length is 64, thesecond tile in the output channel dimension only processes 32 rows which is only half loaded. Inthis case, vector length is set to 32 to maximize computation resource utilization. The V selectionstrategy here is to maximize V while utilizing shared memory resources as much as possible. All the results in this table use the same ﬁne-tuning process.
We ﬁne-tune each network for 40 epochsafter pruning with the same learning rate of 0.0008.
Also, each layer can hold different sparsity ratiocredit to our acceleration for convolution at low sparsity ratio.
[Table]
[Table]
5.2 Convolution Kernel Speedup
As shown in Fig 4, we evaluate the speedup of our method on three popular CNN models.
We use thecuDNN convolution operator as the dense baseline.
The ﬁrst three graphs in Fig 4 represent threetypical convolution shapes in CNN models: small channel size with a large feature map, mediumchannel size with a medium feature map, and large channel size with a small feature map.
Our kernelcan accelerate all these types of convolution layers while exceeds at the twelfth convolution layer of
Resnet50 which is the most used kind of convolution layer 4 .
8 × and 3 .
87 × over cuDNN on V100 at80% sparsity, vector lengths 64 and 32.
5.3 Comparing Different Sparsity Patterns
We replicate two vector-level sparsity pattern, balanced sparsity(NVIDIA 2:4)[15] and Shfl _ BW[9] for comparison.
Other vector-level sparsity patterns such as Tile-wise[5] are slower than theformer two patterns.
Also, these patterns lack implementation for convolution.
Table 2 shows results of Resnet50 on ImageNet directly copied from Shfl _ BW paper, where anexpensive method Grow and Prune[12] is used to recover its accuracy.
Grow and Prune is a sparsitypattern independent method.
We ﬁne-tune our pretrained Resnet50 model for 20 epochs with thelearning rate of 0.001.
We lower the sparsity of our network to 70% where our method demonstrates73.35% top1 accuracy and 2 . 79 × speedup.
Our method exhibits an obviously better speed-accuracytradeoff compared to the Shﬂ_BW.
The OVW pattern also achieves a better speedup compared tobalanced sparsity while recovering the full accuracy of the original model.
To ensure fair comparison, we reproduce these results under the same setting on Resnet50 and Cifar100, as shown in Fig 5.
We ﬁne-tune each network for 40 epochs after pruning from pretraineddense models with the same learning rate of 0.0008.
The OVW pattern dominates the speed-accuracytrade-off in vector-level sparsity.
Conclusion
Accelerating sparse convolution poses a greater challenge than accelerating sparse matrix multiplication.
In this work, we propose a novel sparsity pattern, the OVW pattern to facilitate the sparseconvolution acceleration under intact accuracy.
Although, limitations exist that our method reliesheavily on the hardware supports for the implicit GEMM convolution algorithm.
The performance ofour base dense kernel against the commodity unpublished ones is unstable.
Our method does notacquire the same amount of acceleration rate on matrix multiplication and the acceleration rate ofour method is subject to ﬁlter shape.
Its performance also degrades in specialized convolution layerswhere data reusage opportunity is limited.
In consideration of this, our GPU implementation stilllargely outperforms all sparse acceleration approaches exceedingly with a sparse pattern of similarﬂexibility.
We hope this work can ﬁll the vacancy of specialized sparse convolution kernel design andour methodology can inspire further research in this domain.