Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. In this paper, we investigate a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose those channels that really contribute to discriminative power. To this end, we introduce additional discrimination-aware losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels outperforms the baseline model by 0.39% in top-1 accuracy. Recent studies on model compression mainly contain three categories, namely, quantization [31, 53], sparse or low-rank compressions [8, 9] , and channel pruning [25, 26, 48, 50] . Network quantization seeks to reduce the model size by quantizing float weights into low-bit weights (e.g., 8 bits or even 1 bit). However, the training is very difficult due to the introduction of quantization errors. Making sparse connections can reach high compression rate in theory, but it may generate irregular convolutional kernels which need sparse matrix operations for accelerating the computation. In * Authors contributed equally. Network quantization. In [31], Rastegari et al. propose to quantize parameters in the network into +1/ − 1. The proposed BWN and XNOR-Net can achieve comparable accuracy to their full-precision counterparts on large-scale datasets. In [54] , high precision weights, activations and gradients in CNNs are quantized to low bit-width version, which brings great benefits for reducing resource requirement and power consumption in hardware devices. By introducing zero as the third quantized value, ternary weight networks (TWNs) [21, 55] can achieve higher accuracy than binary neural networks. Explorations on quantization [53] show that quantized networks can even outperform the full precision networks when quantized to the values with more bits, e.g., 4 or 5 bits. Sparse or low-rank connections. To reduce the storage requirements of neural networks, Han et al. suggest that neurons with zero input or output connections can be safely removed from the network [10] . With the help of the 1 / 2 regularization, weights are pushed to zeros during training. Subsequently, the compression rate of AlexNet can reach 35× with the combination of pruning, quantization, and Huffman coding [9] . Considering the importance of parameters is changed during weight pruning, Guo et al. propose dynamic network surgery (DNS) in [8] . Training with sparsity constraints [37, 46] has also been studied to reach higher compression rate. Deep models often contain a lot of correlations among channels. To remove such redundancy, low-rank approximation approaches have been widely studied [4, 5, 17, 36] . For example, Zhang et
Discrimination-aware Channel Pruning (DCP) REF selects channels based on their discriminative power.
Discrimination-aware Channel Pruning for Deep Neural Networks
ArXiv
ArXiv
Computer Science