Knowledge Distillation Based on Pruned Model The high computational complexity of deep neural networks makes them challenging to deploy in practical applications. Recent efforts mainly involve pruning and compression the weights of layers to reduce these costs, and use randomly initializing weights to finetune the pruned model. However, these approaches always lose important weights, resulting in the compressed model performing that is even worse than the original model. To address this problem, we propose a novel method replaced the traditional finetuning method with the knowledge distillation algorithm in this paper. Meanwhile, With the Resnet152 model, our method obtained the accuracy of 73.83% on CIFAR100 data and 22x compression, respectively, ResNet110 SVHN achieve 49x compression with 98.23% accuracy and all of which are preferable to the stateoftheart.

Identifying and Pruning Redundant Structures for Deep Neural Networks Deep convolutional neural networks have achieved considerable success in the field of computer vision. However, it is difficult to deploy stateoftheart models on resourceconstrained platforms due to their high storage, memory bandwidth, and computational costs. In this paper, we propose a structured pruning method which employs a threestep process to reduce the resource consumption of neural networks. First, we train an initial network on the training set and evaluate it on the validation set. Next, we introduce an iterative pruning and finetuning algorithm to identify and prune redundant structures, which results in a pruned network with a compact architecture. Finally, we train the pruned network from scratch on both the training set and validation set to obtain the final accuracy on the test set. In the experiments, our pruning method significantly reduces the model size (by 87.2% on CIFAR10), saves inference time (53.3% on CIFAR10), and achieves better performance as compared to recent stateoftheart methods.