Rigging the Lottery: Making All Tickets Winners
Authors: Utku Evci, Erich Elsen, Pablo Castro, Trevor Gale
Keywords: sparse training, sparsity, pruning, lottery tickets, imagenet, resnet, mobilenet, efficiency, optimization, local minima
TL;DR: We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset.
Abstract: Sparse neural networks have been shown to yield computationally efficient networks with improved inference times.  There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017;Zhu & Gupta, 2018; Louizos et al., 2017; Li et al., 2016; Guo et al., 2016).  This limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires less floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset. Finally,  we  provide  some  insights  into  why  allowing  the  topology  to change during the optimization can overcome local minima encountered when the topology remains static.