  Training sparse networks to converge to the same performance as dense neural
architectures has proven to be elusive. Recent work suggests that
initialization is the key. However, while this direction of research has had
some success, focusing on initialization alone appears to be inadequate. In
this paper, we take a broader view of training sparse networks and consider the
role of regularization, optimization and architecture choices on sparse models.
We propose a simple experimental framework, Same Capacity Sparse vs Dense
Comparison (SC-SDC), that allows for fair comparison of sparse and dense
networks. Furthermore, we propose a new measure of gradient flow, Effective
Gradient Flow (EGF), that better correlates to performance in sparse networks.
Using top-line metrics, SC-SDC and EGF, we show that default choices of
optimizers, activation functions and regularizers used for dense networks can
disadvantage sparse networks. Based upon these findings, we show that gradient
flow in sparse networks can be improved by reconsidering aspects of the
architecture design and the training regime. Our work suggests that
initialization is only one piece of the puzzle and taking a wider view of
tailoring optimization to sparse networks yields promising results.
