  Network pruning is an effective approach to reduce network complexity without
performance compromise. Existing studies achieve the sparsity of neural
networks via time-consuming weight tuning or complex search on networks with
expanded width, which greatly limits the applications of network pruning. In
this paper, we show that high-performing and sparse sub-networks without the
involvement of weight tuning, termed "lottery jackpots", exist in pre-trained
models with unexpanded width. For example, we obtain a lottery jackpot that has
only 10% parameters and still reaches the performance of the original dense
VGGNet-19 without any modifications on the pre-trained weights. Furthermore, we
observe that the sparse masks derived from many existing pruning criteria have
a high overlap with the searched mask of our lottery jackpot, among which, the
magnitude-based pruning results in the most similar mask with ours. Based on
this insight, we initialize our sparse mask using the magnitude pruning,
resulting in at least 3x cost reduction on the lottery jackpot search while
achieves comparable or even better performance. Specifically, our
magnitude-based lottery jackpot removes 90% weights in the ResNet-50, while
easily obtains more than 70% top-1 accuracy using only 10 searching epochs on
ImageNet.
