Real-time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks: feature map-wise, kernel-wise, and intra-kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, in parallel computing environments, and in hardware-based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by assessing the misclassification rate with a corresponding connectivity pattern. The pruned network is retrained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra-kernel strided sparsity with a simple constraint can significantly reduce the size of the kernel and feature map tensors. The proposed work shows that when pruning granularities are applied in combination, we can prune the CIFAR-10 network by more than 70% with less than a 1% loss in accuracy. goal can be achieved by applying the pruning technique to high-performance large networks, where pruning reduces the computational cost. Furthermore, these lightweight networks can be implemented using only on-chip memory for energy savings because frequent DRAM accesses consume much energy. Pruning induces sparsity in a network and can be categorized as structured and unstructured. Unstructured pruning does not follow a specific geometry or constraint. In most cases, this technique needs extra information to represent sparse locations. It leads to irregular sparsity, which is difficult to exploit for efficient computation. On the other hand, structured sparsity places non-zero parameters at well-defined locations. This kind of constraint enables modern CPUs and Graphics Processing Units (GPUs) to easily exploit computational savings. Network pruning has been studied by several researchers [Han et al. 2015a [Han et al. , 2015b Castellano et al. 1997; Collins and Kohli 2014; Stepniewski and Keane 1997; Reed 1993] . The works of Han et al. [2015a Han et al. [ , 2015b have shown that a much bigger portion of weights can be set to zero with minimum or no loss in performance. They train a network with an additional L1/L2 loss function on the weights and gradually prune it. If the weight of a connection is less than a threshold, the connection is dropped. The authors in Han et al. [2015b] further extend this work by quantizing the finally pruned network [Han et al. 2015a ]. However, both works have to explicitly locate nonzero weights with sparse representation. Conventionally, sparse representation uses the Compressed Sparse Row/Compressed Sparse Column (CSR/CSC) format, which represents m non-zero numbers with 2m + n + 1 numbers, where n represents the number of rows or columns. The work of Han et al. [2015a] shows that half of the AlexNet memory space is required for storing the indices of the non-zero parameters. This also doubles memory accesses because each weight fetch now becomes an (index, weight) pair. Our proposed work does not demand such extra representation. In this work, we explore feature map and intra-kernel sparsities as a means of structured pruning. In the feature map pruning, all incoming and outgoing weights to/from a feature map are pruned. The intra-kernel sparsity prunes weights in a kernel. The kernel-level pruning is a special case of intra-kernel sparsity with 100% pruning. These pruning granularities can be applied in various combinations and different orders. The proposed work contains the following contributions:
Intra-kernel strided pruning is one case investigated in the work of Anwar et al. REF .
Structured Pruning of Deep Convolutional Neural Networks
JETC
Computer Science Mathematics