K EEP THE G RADIENTS F LOWING : U SING G RADIENT F LOW TO S TUDY S PARSE N ETWORK O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive.
Recent work suggests that initialization is the key.
However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate.
In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks.
We measure the gradient ﬂow across different networks and datasets and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks.
Based upon these ﬁndings, we show that gradient ﬂow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime.
Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results
I NTRODUCTION
Over the last decade, a “bigger is better” race in the number of model parameters has gripped the ﬁeld of machine learning (Amodei et al., 2018;
Thompson et al., 2020), primarily driven by overparameterized deep neural networks (DNNs).
Additional parameters improve top-line metrics, but drive up the cost of training (Horowitz, 2014; Strubell et al., 2019; Hooker, 2020) and increase the latency and memory footprint at inference time (Warden & Situnayake, 2019; Samala et al., 2018;
Lane & Warden, 2018).
Moreover, overparameterized networks have been shown to be more prone to memorization (Zhang et al., 2016).
To address some of these limitations, there has been a renewed focus on compression techniques that preserve top-line performance while improving efﬁciency.
A large amount of research focus has centered on pruning, where weights estimated to be unnecessary are removed from the network at the end of training (Louizos et al., 2017; Wen et al., 2016;
Cun et al., 1990; Hassibi et al., 1993a;
Str¨om, 1997; Hassibi et al., 1993b; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).
Pruning has shown a remarkable ability to preserve top-line metrics of performance, even when removing the majority of weights (Hooker et al., 2019; Gale et al., 2019).
However, most pruning techniques still require training a large, overparameterized model before pruning a subset of weights.
Due to the drawbacks of starting dense prior to introducing sparsity, there has been a recent focus on methods that allow networks which start sparse at initialization, to converge to similar performance as dense networks (Frankle & Carbin, 2018; Frankle et al., 2019b; Liu et al., 2018a).
These efforts have focused disproportionately on trying to understand the properties of initial sparse weight distributions that allow for convergence.
However, while this work has had some success, focusing on initialization alone has proven to be inadequate (Frankle et al., 2020;
Evci et al., 2019).
In this work, we take a broader view of why training sparse networks to converge to the same performance as dense networks has proven to be elusive.
We reconsider many of the basic building blocks of the training process and ask whether they disadvantage sparse networks or not.
Our work focuses on the behaviour of networks with random, ﬁxed sparsity at initialization and we aim to gain further intuition into how these networks learn.
Furthermore, we provide tooling tailored to the analysis of these networks.
In order to effectively study sparse network optimization in a controlled environment, we propose an experimental framework, Same Capacity Sparse vs Dense Comparison ( SC-SDC ).
Contrary to most prior work comparing sparse to dense networks, where overparameterized dense networks are compared to smaller sparse networks, SC-SDC compares sparse networks to their equivalent capacity dense networks (same number of active connections and depth).
This ensures that the results are a direct result of sparse connections themselves and not due to having more or fewer weights (as is the case when comparing large, dense networks to smaller, sparse networks).
We go beyond simply comparing top-line metrics by also measuring the impact on gradient ﬂow of each intervention.
Historically, exploding and vanishing gradients were a common problem in neural networks (Hochreiter et al., 2001; Hochreiter, 1991; Bengio et al., 1994; Glorot & Bengio, 2010; Goodfellow et al., 2016).
Recent work has suggested that poor gradient ﬂow is an exacerbated issue in sparse networks (Wang et al., 2020;
Evci et al., 2020).
To accurately measure gradient ﬂow in sparse networks, we propose a normalized measure of gradient ﬂow, which we term Effective
Gradient Flow ( EGF ) – this measure normalizes by the number of active weights and thus is better suited to studying the training dynamics of sparse networks.
We use this measure in conjunction with SC-SDC , to see where sparse optimization fails and to consider where this failure could be a result of poor gradient ﬂow.
Contributions Our contributions can be enumerated as follows:
Measuring effective gradient ﬂow We conduct large scale experiments to evaluate the role of regularization, optimization and architecture choices on sparse models.
We evaluate multiple datasets and architectures and propose a new measure of gradient ﬂow, Effective Gradient Flow ( EGF ), that we show to be a stronger predictor of top-line metrics such as accuracy and loss than current gradient ﬂow formulations.
Not all optimizers and regulizers are created equal Weight decay and data augmentation can hurt sparse network optimization, particularly when used in conjunction with accelerating, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam (Kingma & Ba, 2014) and RMSProp (Hinton et al., 2012).
We show this is highly correlated to a high EGF (gradient ﬂow).
Batch normalization plays a disproportionate role in stabilizing sparse networks We show that batch normalization is more important for sparse networks than it is for dense networks, which suggests that gradient instability is a key obstacle to starting sparse.
Changing activation functions can beneﬁt sparse networks We benchmark a wide set of activation functions, speciﬁcally ReLU (Nair & Hinton, 2010) and non-saturating activation functions such as PReLU (He et al., 2015), ELU (Clevert et al., 2015), SReLU (Jin et al., 2015),Swish (Ramachandran et al., 2017) and Sigmoid (Neal, 1992).
Our results show that when using adaptive optimization methods, Swish is a promising activation, while when using stochastic gradient descent, PReLU preforms better than the other activation functions.
Implications Our work is timely as sparse training dynamics are poorly understood.
Most training algorithms and methods have been developed to suit training dense networks.
Our work provides insight into the nature of sparse optimization and suggests a wider viewpoint beyond initialization is necessary to converge sparse networks to comparable performance as dense.
Our proposed approach provides a more accurate measurement of the training dynamics of sparse networks and can be used to inform future work on the design of networks and optimization techniques that are tailored explicitly to sparsity.
M ETHODOLOGY 2. S AME C APACITY S PARSE VS D ENSE C OMPARISON
Our goal is to measure what architecture and optimization choices favor sparse networks relative to dense networks.
To fairly compare sparse and dense networks, we propose Same Capacity Sparse vs Dense Comparison ( SC-SDC ), a simple framework which allows us to study sparse network optimization and identify what training conﬁgurations are not well suited for sparse networks.
SC-SDC can be summarized as follows (See Figure 1 for an overview):
Initialize For a chosen network depth (number of layers) L and a maximum network width N MaxW , we compare sparse and dense networks at various widths, while ensuring they have the same paramater count.
Initially, we mask the weights θ S of sparse network S :
[Equation]
, where θ lS (cid:12) m l denotes an element-wise product of the weights θ S of layer l and the random binary matrix (mask) for layer l , m l , a lS is the nonzero weights in layer l of sparse network S and a lD is the nonzero weights in layer l of dense network D (all the weights since no masking occurs).
For a fair comparison, we need to ensure the same number of nonzero weights for sparse network S and dense network D , across each layer L .
[Equation]
We provide more implementation details of how we achieve this in Appendix A.1.
Match active weight distributions Following prior work (Liu et al., 2018b; Gale et al., 2019), we ensure the nonzero weights at initialization of the sparse and dense networks are sampled from the same distribution at each layer as follows:
[Equation]
, where P l refers to the initial weight distribution at layer l , for example Kaiming initialization (He et al., 2015).
This ensures that both sets of active weights (sparse and dense) are initially sampled from the same distribution.
Train We then train the sparse and dense networks for 1000 epochs (allowing for convergence).
Evaluate the better architecture We gather the results across the widths/capacity levels and conduct a paired, one-tail Wilcoxon signed-rank test (Wilcoxon, 1945) to evaluate the better architecture.
Our null hypothesis ( H 0 ) is that sparse networks have similar or worse test accuracy than dense networks (lower or the same median), while our alternative hypothesis ( H 1 ) is that sparse networks have better test accuracy performance than dense networks of the same capacity (higher median).
This can be formulated as:
[Equation]
Our goal of SC-SDC is to compare sparse and dense networks at the same capacity level.
By same capacity, we are referring to the same number of active weights, but other notions of same capacity can also be used.
We brieﬂy discuss this in Appendix A.1.
[Table]
[Table]
We compare the average absolute Kendall Rank correlation between different formulations of gradient ﬂow and generalization.
The subscript denotes the p -norm ( l 1 or l 2 norm).
We see that
EGF has higher absolute correlation when compared to standard gradient ﬂow measures.
We also see that is consistent across Fashion MNIST (see Appendix B.1)
[Table]
2. M EASURING G RADIENT F LOW
Gradient ﬂow (GF) is used to study optimization dynamics and typically approximated by taking the norm of the gradients of the network (Pascanu et al., 2013;
Nocedal et al., 2002; Chen et al., 2018;
Wang et al., 2020; Evci et al., 2020).
We consider a feedforward neural network f : R D → R , with function inputs x ∈ R D and network weights θ .
The gradient norm is usually computed by concatenating all the gradients of a network into a single vector, g = ∂ C ∂ θ , where C is our cost function.
Then the vector norm is taken as follows:
[Equation]
where p denotes the pth-norm.
Effective Gradient Flow Traditional measures of gradient ﬂow take the l 1 or l 2 norm of all the gradients (Chen et al., 2018;
Pascanu et al., 2013; Evci et al., 2020).
This is not appropriate for sparse networks, as this would include gradients of masked weights which have no inﬂuence on the forward pass.
Furthermore, computing l 1 or l 2 across all weights in the networks gives disproportionate inﬂuence to layers with more weights.
We instead propose a simple modiﬁcation of Equation 5, which we term Effective Gradient Flow ( EGF ), that computes the average, masked gradient (only gradients of active weights) norm across all layers.
We calculate EGF as follows:
[Equation]
where L is number of layers and ∂ C ∂θ l (cid:12) m l denotes an element-wise product of the gradients of layer l , ∂ C ∂θ l , and the mask m l applied to the weights of layer l .
For a fully dense network, m l is a matrix of all ones, since no gradients are masked.
EGF has the following favourable properties:
• Gradient ﬂow is evenly distributed across layers EGF distributes the gradient norm across the layers equally, preventing layers with a lot of weights from dominating the measure and also preventing layers with vanishing gradients from being hidden in the formulation, as is the case with equation 5 (when all gradients are appended together).
• Only gradients of active weights are used EGF ensures that for sparse networks, only gradients of active weights are used.
Even though weights are masked, their gradients are not necessarily
[Table]
zero since the partial derivative of the weight wrt.
the loss, is inﬂuenced by other weights and activations.
Thereby a weight can be zero, but its gradient can be nonzero.
• Possibility for application in gradient-based pruning methods Tanaka et al
2020) showed that gradient-based pruning methods like GRASP
(Wang et al., 2020) and SNIP (Lee et al., 2018a ), disproportionately prune large layers and are susceptible to layer-collapse, which is when an algorithm prunes all the weights in a speciﬁc layer.
Due to the fact that EGF is evenly distributed across layers, maintaining EGF (as opposed to standard gradient norm) could possibly be used as a pruning criteria.
Furthermore, current approaches measuring or approximating the change in gradient ﬂow during pruning in sparse networks (Wang et al Evci et al
., 2020; Singh Lubana & Dick, 2020), could beneﬁt from this new formulation
To evaluate EGF against other standard gradient norm measures, such as the l 1 and l 2 norm, we empirically compare these measures and their correlation to test loss and accuracy.
We take the absolute average of the Kendall Rank correlation (Kendall, 1938), across the different experiment conﬁgurations.
We follow a similiar approach to Jiang et al.
, but unlike their work which has focused on correlating network complexity measures to the generalization gap, we measure the correlation of gradient ﬂow to performance (accuracy and loss).
We measure gradient ﬂow atpoints evenly spaced throughout training, speciﬁcally at the end of epoch 0, 99. 199, 299, 399, 499, 599, 699, 799, 899 and 999.
Our results from Table 1 shows that EGF has a higher average absolute correlation to both test loss and accuracy.
This is also true of Fashion MNIST (see Appendix B.1).
Due to the comparative beneﬁts of EGF , we use it for the remainder of the paper to measure the impact of interventions.
We include all measures of gradient ﬂow in Appendix B for completeness.
2. A RCHITECTURE , N ORMALIZATION , R EGULARIZATION AND O PTIMIZER V ARIANTS
We brieﬂy describe our key experiment variants below, and also include for completeness all unique variants in Table 2.
Activation functions ReLU networks (Nair & Hinton, 2010) are known to be more resilient to vanishing gradients than networks that use Sigmoid or Tanh activations, since they only result in vanishing gradients when the input is less than zero, while on active paths, due to ReLU’s linearity, the gradients ﬂow uninhibited (Glorot et al., 2011).
Although most experiments are run on ReLU networks, we also explore different activation functions, namely PReLU (He et al., 2015), ELU (Clevert et al., 2015), SReLU (Jin et al., 2015) and Sigmoid (Neal, 1992).
Batch normalization and Skip Connections Other methods to help alleviate the vanishing gradient problem include the addition of skip connections (every two layers) (Srivastava et al., 2015; He et al., 2016) and batch normalization (Ioffe & Szegedy, 2015).
We empirically explore these methods.
Optimization and Regularization techniques We explore the impact of popular regularization methods: weight decay/ l 2 regularization (0.0001) (Krogh & Hertz, 1992; Hanson & Pratt, 1989) and data augmentation (random crops and random horizontal ﬂipping (Krizhevsky et al., 2012)).
Furthermore, we benchmark the impact of the most widely used optimizers such as minibatch stochastic gradient descent (with momentum (0.9) (Sutskever et al., 2013; Polyak, 1964) and without momentum (Robbins & Monro, 1951)) , Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and
RMSProp (Hinton et al., 2012).
E MPIRICAL S ET - UP
SC-SDC MLP Setting We use the SC-SDC empirical setting (section 2.1) for all experiment variants.
We train over 6000 MLPs for 1000 epochs and evaluate performance on CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009).
We compare sparse and dense networks across various widths, depths, learning rates, regularization and optimization methods as shown in Table 2.
We choose a max network width N MaxW of n + 4 , where n is the input dimension of the network.
In the case of CIFAR, n = 3072 and so our maximum width N MaxW = 3076 .
We repeat these experiments with one, two and four hidden layers, with the number of active weights in these networks ranging from 949 , 256 to 31 , 765 , 568 weights.
In section 4, we discuss results achieved using four hidden layers and we provide the one and two hidden layer results in Appendix C.
Dense Width Following from SC-SDC , these networks are compared at various network widths, speciﬁcally a width of 308, 923, 1538, 2153, 2768 (10%, 30%, 50%, 70% and 90% of our maximum width N MaxW (3076)) as shown in Table 2.
We use the term dense width to refer to the width of a network if that network was dense.
For example, when comparing sparse and dense networks at a dense width of 308, this means the dense network has a width of 308, while the sparse network has a width of N MaxW (3076), but has the same number of active connections as the dense counterpart.
We provide more detailed discussion of the choices made in the SC-SDC implementation in
Appendix A.1.
Extended CNN Setting We also extend our experiments to Wide Resnet-50 (the WRN-28-10 variant) (Zagoruyko & Komodakis, 2016) and use the optimization and regularization conﬁgurations from the paper.
R ESULTS AND D ISCUSSION 4. C OMPARISON OF DENSE AND SPARSE INTERVENTIONS USING SC-SDC
In this section, we use the results of the Wilcoxon signed rank test from SC-SDC to identify where optimization choices are currently well suited for sparse networks and which are not.
Futhermore, for each variant, we also measure the gradient ﬂow using EGF 1 as described in the previous section.
Our main ﬁndings show that:
Batch normalization is critical to training sparse networks, more so than it is for dense networks.
This suggests that gradient instability is a key obstacle for sparse optimization.
Weight decay and data augmentation can hurt both sparse and dense network optimization. This particularly occurs when used with accelerating, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam and RMSProp (Ruder, 2016). In these methods, large gradient ﬂow ( EGF ) strongly correlates to poor test accuracy.
We show that batch normalization helps alleviate this issue. Non-saturating activation functions, such as Swish (Ramachandran et al., 2017) and PReLU (He et al., 2015), achieve promising results in both the sparse and dense regime. However, these results are more apparent when sparse networks are used and so this could motivate for the use of similiar activation functions for training sparse networks. Batch normalization plays a disproportionate role in stabilizing sparse networks Batch normalization ensures that the distribution of the nonlinearity inputs remains stable as the network trains, which was hypothesized to help stabilize gradient propagation (gradients do not explode or vanish) (Ioffe & Szegedy, 2015).
Following from Table 3a and 3b, we see that batch normalization is statistically more important for sparse networks then dense networks, across most conﬁgurations and learning rates. From Table 3a and Figure 12, we see methods such as L2 and data augmentation favour dense networks.
However, with the addition of batch normalization, these methods favour sparse variants. This is especially apparent in Figure 12, where for adaptive gradient methods such as Adam and RMSProp, batch normalization improves performance across all sparse optimizers, while resulting [Table]
Relu
Swish
PRelu SRelu Sigmoid Elu
[Table]
[Table] [Table] in a lower, more stable gradient ﬂow. This further emphasis the importance of stabilizing gradient ﬂow, particularly in sparse networks.
Weight Decay and Data Augmentation can hurt sparse network optimization From Table 3a, we can see that weight decay and data augmentation, with most optimizers, do not favour sparse networks over dense networks. This behaviour changes when batch normalization is introduced, as we can see from Table 3a and Table 3b, further emphasizing the importance of batch normalization in the sparse regime. When we take a closer look at the effects of these regularization methods on sparse network accuracy (Figure 2,Figure 13)
[Table] we see that weight decay (even with batch normalization) drastically decrease accuracy when used with adaptive optimization methods that use an exponentially decaying average of past squared gradients (Adam and RMSProp). Furthermore, it results in distinctively larger EGF values, which hints at Adam and RMSProp being more sensitive to larger gradient norms than other optimizers. This agrees with Loshchilov & Hutter (2017)’s work which proposed a different formulation of weight decay for adaptive methods, since the current L 2 regularization for adaptive methods could lead to weight with large gradients being regularized less, although this was not experimentally veriﬁed.
In the context of data augmentation, we see little to no improvement in test accuracy for dense or sparse variants. We believe this is because data augmentation methods have been designed for convolutional networks (CNNs) and not for multilayer perceptrons (MLPs). For MLPs, DA might not have the intended effect since MLPs work with a ﬂattened image, across all the channels.
This means random crops and ﬂips might not have the same effect as they do for CNNs, which leverage pooling and 3d conv layers. [Figure] NR - No Regularization, BN - Batchnorm, SC - Skip Connections, DA - Data Augmention and L2- weight decay.
The potential of non-saturating activation functions - Swish and PReLU We also explore the effect of different activation functions on sparse network optimization.
For the activation function variants, the best conﬁguration for each optimizer was chosen.
For Adagrad, Adam and RMSProp we use BN, SC and DA, while for SGD, we use BN, SC, L2 and DA.
From Table 3c, we see that Swish, PReLU and Sigmoid favour sparse architectures, but from the performance results from Figure 11a, we see that only Swish and PReLU are viable activation choices.
We continue to see a consistent trend for adaptive methods (most notably in Adam and RMSProp), that higher EGF values, for example in SReLU, correspond to poor performance (Figure 11b), while promising methods result in a lower EGF value (such as Swish). This further emphasizes how EGF can be used to guide advances in network optimization.
4. G ENERALIZATION OF RESULTS ACROSS ARCHITECTURE TYPES .
In this section, we move on from SC-SDC and extend our result to WideResnet-50.
It is important to note that in Section 4.1, we tested the effect of interventions in isolation, while in this section we test the effect of certain intervention together.
We note from Figure 3, that most of our results from
SC-SDC also hold on larger, more complicated models.
We see that L 2 regularization hurts performance for adaptive methods (Adagrad and Adam) and also results in higher EGF values (Figure 14).
Furthermore, we also that Swish is a promising activation function for adaptive methods and leads to lower EGF . This shows that the SC-SDC results are not constrained to small scale experiments and that it can be used to learn about dynamics of larger, more complicated networks.
R ELATED WORK
Pruning at Initialization Methods that prune at initialization aim to start sparse, instead of ﬁrst pre-training an overparameterized network and then pruning.
These methods use certain criteria to estimate at initialization, which weights should remain active.
This criteria includes using the connection sensitivity (Lee et al., 2018b), gradient ﬂow (via the Hessian vector product) (Wang et al., 2020) and conversation of synaptic saliency (Tanaka et al., 2020).
Another branch of pruning is Dynamic Sparse Training, which uses information gathered during the training process, to dynamically update the sparsity pattern of these sparse networks (Mostafa & Wang, 2019; Bellec et al., 2017;
Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019;
Evci et al., 2019)
While our work is motivated by the same goal of allowing networks to start sparse and converge to the same performance as dense networks, we instead focus on the impact of optimization and regularization choices on sparse networks.
Sparse Network Optimization as Pruning Criteria Optimization in sparse networks has often been neglected in favour of studying network initialization.
However, there has been work that has looked at sparse network optimization from different perspectives, mainly as a guide for pruning criteria.
This includes using gradient information (Mozer & Smolensky, 1989; LeCun et al., 1989;
Hassibi & Stork, 1992; Karnin, 1990), approximates of gradient ﬂow (Wang et al., 2020; Dettmers & Zettlemoyer, 2019;
Evci et al., 2020) and Neural Tangent Kernel (NTK) (Liu & Zenke, 2020) to guide the introduction of sparsity.
Sparse Network Optimization to study Network Dynamics Apart from use as pruning criteria, optimization information has been used to investigate aspects of sparse networks, such as their loss landscape (Evci et al., 2019), how they are impacted by SGD noise (Frankle et al., 2019a), the effect of different activation functions (Dubowski, 2020) and their weight initialization (Lee et al., 2019).
Our work differs from these approaches as we consider more aspects of the optimization and regularization process in a controlled experimental setting (SC-SDC), while using EGF to reason about some of the results.
C ONCLUSION AND F UTURE W ORK
In this work, we take a wider view of sparse optimization strategies and introduce appropriate tooling to measure the impact of architecture and optimization choices on sparse networks ( EGF , SC-SDC ).
Our results show that weight decay and data augmentation can hurt optimization, when adaptive optimization methods are used and this usually corresponds to a much higher EGF .Furthermore, we show how batch normalization is critical to training sparse networks, more so than it is for dense networks as it helps stabilize gradient ﬂow.
Lastly, we also show the potential of non-saturating activation functions for sparse networks such as Swish and PReLU.
A SC-SDC
In this section, we provide more information about SC-SDC and its beneﬁts.
A. SC-SDC IMPLEMENTATION DETAILS
Wilcoxon Signed Rank Test
This is a non-parametric test that compares dependent or paired samples, without assuming the differences in between the paired experiments are normally distributed (McDonald, 2009; Demˇsar, 2006).
Random Sparsity Our work focuses on the training dynamics of random, sparse networks.
This ensures that what is learned is not dependent on a speciﬁc pruning method, but rather can be used to better understand sparse training in general.
Going forward, it would be interesting to explore these dynamics on pruned networks.
We achieve random sparsity, by generating a random mask for each layer and then multiply the weights by this mask during each forward pass.
The sparsity is distributed evenly across the network.
For example, a 20% sparse MLP has 20% of the weights remaining in each layer.
Dense Width A critical component to how we specify our experiments is a term we deﬁne as dense width.
In order to fairly compare sparse and dense networks, we need them to have the same number of active connections at each depth.
In the case of sparse networks, this means ensuring they have the same number of active connections as the dense networks, while remaining sparse.
Dense width refers to the width of a network if that network was dense.
This process of comparing sparse and dense networks at different dense widths is illustrated in ﬁgure 5.
Fair comparison of Sparse and Dense networks As can be seen from ﬁgure 5, SC-SDC ensures the exact same active parameter count, but the sparse networks will be connected to more neurons.
It is possible that the increased number of activations being used can lead to sparse networks having higher representational power, however most work on expressivity of neural networks looks at this from a depth perspective and proves certain depths of networks are universal approximators (Eldan & Shamir, 2016;
Hornik et al., 1989; Funahashi, 1989).
To this end, we ensure these networks have the same depth, but we believe going forward an interesting direction would be ensuring they have a similar amount of active neurons.
SC-SDC comparison details For completeness, we provide more details of how we ensure sparse and dense networks are of the same capacity.
Following from equation 2, to ensure the same number of weights in sparse and dense networks, we can ensure they have the same number of active weights at each layer as follows:
[Equation]
This is achieved by masking each of the weight layers of sparse network S :
[Equation]
, where m l is a random binary matrix (mask) for layer l , s.t.
||
m l ||
= a lD , where a lD is determined by the chosen capacity, these networks will be compared at.
For SC-SDC , we need a maximum network width N MaxW and comparison width N W .
We choose a max network width N MaxW of n + 4 , where n is the input dimension of the network.
In the case of CIFAR, n = 3072 and so our maximum width N MaxW = 3076 .
The choice of n + 4 follows from Lu et al.
(2017), where the authors prove a universal approximation theorem for width-bounded
ReLU networks, with width bounded to n + 4 .
Our comparison width, N W , is equivalent to dense widths we vary in our experiments - 308, 923, 1538, 2153, 2768.
The dimensions of each of layers are as follows:
First Layer:
[Equation]
Intermediate Layers
[Equation]
Final Layer
[Equation]
, where N MaxW is maximum width of the sparse layer, N W is the comparison width, I is the input dimension, O is output dimension, L is the number of layers in the network, θ lS is the weights in layer l of sparse network S and θ lD is the weights in layer l of dense network D .
This process would be the same for convolutional layers, but there would be a third dimension to handle the different channels.
In ﬁgure 4, we provide an illustrative example showing how to ensure sparse and dense networks are compared fairly.
A. B ENEFITS
The beneﬁts of SC-SDC can be summarized as follows:
• We can better understand sparse network optimization.
SC-SDC allows us to identify which optimization or regularization methods are poorly suited to sparse networks in a controlled setting, ensuring the results are a direct result of the sparse connections themselves.
Learn at what parameter and size budget, sparse networks are better than dense.
Comparing sparse and dense networks of the same capacity allows us to see which architecture is better at different conﬁgurations.
In conﬁgurations where sparse architectures perform better, we could exploit advances in sparse matrix computation and storage (Zhao et al., 2018; Merrill & Garland, 2016) to simply default to sparse architectures.
B G RADIENT F LOW B. EGF RESULTS ON FMNIST
We extend our experiments to Fashion MNIST (Xiao et al., 2017), a dataset that is distinctively different to the CIFAR datasets we used in section 2.2. We ran 450 experiments with networks with four hidden layers, using 0.001 as a learning rate and for 500 epochs. We varied conﬁgurations as follows:
• Optimizers - Adagrad, Adam and SGD with momentum. • Regularization methods - no regularization, batchnorm, skip connections, l2 (0.0001) and data augmentation. From table 4, we see that out of the gradient ﬂow formulations, EGF still correlates better to generalization performance. For completeness, we present the full set of results using the different formulations of gradient ﬂow on CIFAR-100.
Namely, we show ||
g || 1 (5) (Figure 6) ,
||
g || 2 (5) (Figure 7) , egf 1 (6) (Figure 8) and egf 2 (6) (Figure 9). [Table]
C D ETAILED R ESULTS FOR SC-SDC
In this section we presented the detailed results for our experiments.
Detailed Results with a low learning rate (0.001).
Detailed Results with a high learning rate (0.1).
Results for different activation functions.
[Figure]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
Hidden Layers
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]