Reducing the number of neurons of Deep ReLU Networks based on the current theory of Regularization
Authors: Jakob Heiss, Alexis Stockinger, Josef Teichmann
Keywords: Reduction, Compression, Regularization, Theory, Pruning, Deep, Interpretability, Generalization
Abstract: We introduce a new Reduction Algorithm which makes use of the properties of ReLU neurons to reduce significantly the number of neurons in a trained Deep Neural Network. This algorithm is based on the recent theory of implicit and explicit regularization in Deep ReLU Networks from (Maennel et al, 2018) and the authors.

We discuss two experiments which illustrate the efficiency of the algorithm to reduce the number of neurons significantly with provably almost no change of the learned function within the training data (and therefore almost no loss in accuracy).