On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
Authors: Michela Paganini, Jessica Forde
Keywords: Pruning, Lottery Tickets, Science of Deep Learning, Experimental Deep Learning, Empirical Study
TL;DR: Different pruning techniques identify multiple trainable sub-networks within an over-parametrize model, with similar performance but significantly different emergent connectivity structure, weight evolution, and learned functions.
Abstract: We examine how recently documented, fundamental phenomena in deep learn-ing models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics  of  pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of  trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based un-structured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.