K EEP THE G RADIENTS F LOWING : U SING G RADIENT F LOW TO S TUDY S PARSE N ETWORK O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive.
Recent work suggests that initialization is the key.
However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate.
In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks.
We measure the gradient ﬂow across different networks and datasets and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks.
Based upon these ﬁndings, we show that gradient ﬂow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime.
Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results
I NTRODUCTION
Over the last decade, a “bigger is better” race in the number of model parameters has gripped the ﬁeld of machine learning (Amodei et al., 2018;
Thompson et al., 2020), primarily driven by overparameterized deep neural networks (DNNs).
Additional parameters improve top-line metrics, but drive up the cost of training (Horowitz, 2014; Strubell et al., 2019; Hooker, 2020) and increase the latency and memory footprint at inference time (Warden & Situnayake, 2019; Samala et al., 2018;
Lane & Warden, 2018).
Moreover, overparameterized networks have been shown to be more prone to memorization (Zhang et al., 2016).
To address some of these limitations, there has been a renewed focus on compression techniques that preserve top-line performance while improving efﬁciency.
A large amount of research focus has centered on pruning, where weights estimated to be unnecessary are removed from the network at the end of training (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993a; Str¨om, 1997; Hassibi et al., 1993b; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).
Pruning has shown a remarkable ability to preserve top-line metrics of performance, even when removing the majority of weights (Hooker et al., 2019; Gale et al., 2019).
However, most pruning techniques still require training a large, overparameterized model before pruning a subset of weights.
Due to the drawbacks of starting dense prior to introducing sparsity, there has been a recent focus on methods that allow networks which start sparse at initialization, to converge to similar performance as dense networks (Frankle & Carbin, 2018; Frankle et al., 2019b; Liu et al., 2018a).
These efforts have focused disproportionately on trying to understand the properties of initial sparse weight distributions that allow for convergence.
However, while this work has had some success, focusing on initialization alone has proven to be inadequate (Frankle et al., 2020;
Evci et al., 2019).
In this work, we take a broader view of why training sparse networks to converge to the same performance as dense networks has proven to be elusive.
We reconsider many of the basic building blocks of the training process and ask whether they disadvantage sparse networks or not.
Our work focuses on the behaviour of networks with random, ﬁxed sparsity at initialization and we aim to gain further intuition into how these networks learn. Furthermore, we provide tooling tailored to the analysis of these networks.
In order to effectively study sparse network optimization in a controlled environment, we propose an experimental framework, Same Capacity Sparse vs Dense Comparison ( SC-SDC ).
Contrary to most prior work comparing sparse to dense networks, where overparameterized dense networks are compared to smaller sparse networks, SC-SDC compares sparse networks to their equivalent capacity dense networks (same number of active connections and depth).
This ensures that the results are a direct result of sparse connections themselves and not due to having more or fewer weights (as is the case when comparing large, dense networks to smaller, sparse networks).
We go beyond simply comparing top-line metrics by also measuring the impact on gradient ﬂow of each intervention.
Historically, exploding and vanishing gradients were a common problem in neural networks (Hochreiter et al., 2001; Hochreiter, 1991; Bengio et al., 1994; Glorot & Bengio, 2010; Goodfellow et al., 2016).
Recent work has suggested that poor gradient ﬂow is an exacerbated issue in sparse networks (Wang et al., 2020; Evci et al., 2020).
To accurately measure gradient ﬂow in sparse networks, we propose a normalized measure of gradient ﬂow, which we term Effective Gradient Flow ( EGF ) – this measure normalizes by the number of active weights and thus is better suited to studying the training dynamics of sparse networks.
We use this measure in conjunction with SC-SDC , to see where sparse optimization fails and to consider where this failure could be a result of poor gradient ﬂow.
Contributions Our contributions can be enumerated as follows:
Measuring effective gradient ﬂow We conduct large scale experiments to evaluate the role of regularization, optimization and architecture choices on sparse models.
We evaluate multiple datasets and architectures and propose a new measure of gradient ﬂow, Effective Gradient Flow ( EGF ), that we show to be a stronger predictor of top-line metrics such as accuracy and loss than current gradient ﬂow formulations.
Not all optimizers and regulizers are created equal Weight decay and data augmentation can hurt sparse network optimization, particularly when used in conjunction with adaptive optimization methods such as Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSProp (Hinton et al., 2012).
Batch normalization plays a disproportionate role in stabilizing sparse networks We show that batch normalization is more important for sparse networks than it is for dense networks, which suggests that gradient instability is a key obstacle to starting sparse.
Changing activation functions can beneﬁt sparse networks We benchmark a wide set of activation functions, speciﬁcally ReLU (Nair & Hinton, 2010) and non-saturating activation functions such as PReLU (He
et al.,
), ELU (Clevert et al.,
), SReLU Jin et al., and Sigmoid (Neal, 1992).
Our results show that when using adaptive optimization methods, such as Adam, Adagrad and RMSProp, SReLU is a promising activation function for sparse networks as it improves the networks gradient ﬂow and achieves better accuracy when compared to the other activation functions.
Furthermore, we also show that when training sparse networks, using stochastic gradient descent (with and without momentum)(Robbins & Monro, 1951; Sutskever et al., 2013; Polyak, 1964), PReLU preforms better than the other activation functions.
Implications Our work is timely as sparse training dynamics are poorly understood.
Most training algorithms and methods have been developed to suit training dense networks.
Our work provides insight into the nature of sparse optimization and suggests a wider viewpoint beyond initialization is necessary to converge sparse networks to comparable performance as dense.
Our proposed approach provides a more accurate measurement of the training dynamics of sparse networks and can be used to inform future work on the design of networks and optimization techniques that are tailored explicitly to sparsity.
M ETHODOLOGY 2. S AME C APACITY S PARSE VS D ENSE C OMPARISON
Our goal is to measure what architecture and optimization choices favor sparse networks relative to dense networks.
To fairly compare sparse and dense networks, we propose Same Capacity Sparse vs Dense Comparison ( SC-SDC ), a simple framework which allows us to study sparse network optimization and identify what training conﬁgurations are not well suited for sparse networks.
SC-SDC can be summarized as follows (See Figure 1 for an overview):
Initialize For a chosen network depth (number of layers) L and a maximum network width N MaxW , we compare sparse and dense networks at various widths, while ensuring they have the same paramater count.
Initially, we mask the weights θ S of sparse network S :
[Equation]
, where θ lS (cid:12) m l denotes an element-wise product of the weights θ S of layer l and the random binary matrix (mask) for layer l , m l , a lS is the nonzero weights in layer l of sparse network S and a lD is the nonzero weights in layer l of dense network D (all the weights since no masking occurs). For a fair comparison, we need to ensure the same number of nonzero weights for sparse network S and dense network D , across each layer L . [Equation]
We provide more implementation details of how we achieve this in Appendix A.1. Match active weight distributions Following prior work (Liu et al., 2018b; Gale et al., 2019), we ensure the nonzero weights at initialization of the sparse and dense networks are sampled from the same distribution at each layer as follows:
[Equation]
, where P l refers to the initial weight distribution at layer l , for example Kaiming initialization (He et al., 2015).
This ensures that both sets of active weights (sparse and dense) are initially sampled from the same distribution.
Train We then train the sparse and dense networks
for 1000 epochs (allowing for convergence). Evaluate the better architecture We gather the results across the widths/capacity levels and conduct a paired, one-tail Wilcoxon signed-rank test (Wilcoxon, 1945) to evaluate the better architecture.
Our null hypothesis ( H 0 ) is that sparse networks have similar or worse test accuracy than dense networks (lower or the same median), while our alternative hypothesis ( H 1 ) is that sparse networks have better test accuracy performance than dense networks of the same capacity (higher median).
This can be formulated as:
[Equation]
Our goal of SC-SDC is to compare sparse and dense networks at the same capacity level.
By same capacity, we are referring to the same number of active weights, but other notions of same capacity can also be used.
We brieﬂy discuss this in Appendix A.1.
[Table]
We compare the average absolute Kendall Rank correlation between different formulations of gradient ﬂow and generalization.
The subscript denotes the p -norm ( l 1 or l 2 norm).
We see that
EGF has higher absolute correlation when compared to standard gradient ﬂow measures
[Table]
2. M EASURING G RADIENT F LOW
Gradient ﬂow (GF) is used to study optimization dynamics and typically approximated by taking the norm of the gradients of the network (Pascanu et al., 2013;
Nocedal et al., 2002; Chen et al., 2018;
Wang et al., 2020; Evci et al., 2020).
We consider a feedforward neural network f : R D → R , with function inputs x ∈ R D and network weights θ .
The gradient norm is usually computed by concatenating all the gradients of a network into a single vector, g = ∂ C ∂ θ , where C is our cost function.
Then the vector norm is taken as follows:
[Equation]
where p denotes the pth-norm.
Effective Gradient Flow Traditional measures of gradient ﬂow take the l 1 or l 2 norm of all the gradients (Chen et al., 2018;
Pascanu et al., 2013; Evci et al., 2020).
This is not appropriate for sparse networks, as this would include gradients of masked weights which have no inﬂuence on the forward pass.
Furthermore, computing l 1 or l 2 across all weights in the networks gives disproportionate inﬂuence to layers with more weights.
We instead propose a simple modiﬁcation of Equation 5, which we term Effective Gradient Flow ( EGF ), that computes the average, masked gradient (only gradients of active weights) norm across all layers.
We calculate EGF as follows:
[Equation]
where L is number of layers and ∂ C ∂θ l (cid:12) m l denotes an element-wise product of the gradients of layer l , ∂ C ∂θ l , and the mask m l applied to the weights of layer l .
For a fully dense network, m l is a matrix of all ones, since no gradients are masked.
EGF has the following favourable properties: • Gradient ﬂow is evenly distributed across layers EGF distributes the gradient norm across the layers equally, preventing layers with a lot of weights from dominating the measure and also preventing layers with vanishing gradients from being hidden in the formulation, as is the case with equation 5 (when all gradients are appended together).
• Only gradients of active weights are used EGF ensures that for sparse networks, only gradients of active weights are used. Even though weights are masked, their gradients are not necessarily zero since the partial derivative of the weight wrt. the loss, is inﬂuenced by other weights and activations.
Thereby a weight can be zero, but its gradient can be nonzero.
[Table] • Possibility for application in gradient-based pruning methods Tanaka et al. (2020) showed that gradient-based pruning methods like GRASP (Wang et al., 2020) and SNIP (Lee et al., 2018a), disproportionately prune large layers and are susceptible to layer-collapse, which is when an algorithm prunes all the weights in a speciﬁc layer. Due to the fact that EGF is evenly distributed across layers, maintaining EGF (as opposed to standard gradient norm) could possibly be used as a pruning criteria.
Furthermore, current approaches measuring or approximating the change in gradient ﬂow during pruning in sparse networks (Wang et al., 2020; Evci et al., 2020; Singh Lubana & Dick, 2020), could beneﬁt from this new formulation.
To evaluate EGF against other standard gradient norm measures, such as the l 1 and l 2 norm, we empirically compare these measures and their correlation to test loss and accuracy.
We take the absolute average of the Kendall Rank correlation (Kendall, 1938), across the different experiment conﬁgurations.
We follow a similiar approach to Jiang et al.
, but unlike their work which has focused on correlating network complexity measures to the generalization gap, we measure the correlation of gradient ﬂow to performance (accuracy and loss). We measure gradient ﬂow atpoints evenly spaced throughout training, speciﬁcally at the end of epoch 0, 99. 199, 299, 399, 499, 599, 699,
, 899 and Our results from Table 1 shows that EGF has a higher average absolute correlation to both test loss and accuracy.
Due to the comparative beneﬁts of EGF , we use it for the remainder of the paper to measure the impact of interventions.
We include all measures of gradient ﬂow in Appendix B for completeness.
2. A RCHITECTURE , N ORMALIZATION , R EGULARIZATION AND O PTIMIZER V ARIANTS
We brieﬂy describe our key experiment variants below, and also include for completeness all unique variants in Table 2.
There is a total of 30 experiments for each conﬁguration: 5 network widths × three independent training runs (different random seeds) × 2 (sparse and dense variant).
Activation functions ReLU networks (Nair & Hinton, 2010) are known to be more resilient to vanishing gradients than networks that use Sigmoid or Tanh activations, since they only result in vanishing gradients when the input is less than zero, while on active paths, due to ReLU’s linearity, the gradients ﬂow uninhibited (Glorot et al., 2011).
Although most experiments are run on ReLU networks, we also explore different activation functions, namely PReLU (He et al., 2015), ELU (Clevert et al., 2015), SReLU (Jin et al., 2015) and Sigmoid (Neal, 1992).
Batch normalization and Skip Connections Other methods to help alleviate the vanishing gradient problem include the addition of skip connections (every two layers) (Srivastava et al., 2015; He et al., 2016) and batch normalization (Ioffe & Szegedy, 2015).
We empirically explore these methods. Optimization and Regularization techniques We explore the impact of popular regularization methods: weight decay/ l 2 regularization (0.0001) (Krogh & Hertz, 1992; Hanson & Pratt, 1989) and data augmentation (random crops and random horizontal ﬂipping (Krizhevsky et al., 2012)).
Furthermore, we benchmark the impact of the most widely used optimizers such as minibatch stochastic gradient descent (with momentum (0.9) [Table] [Table] This table presents the Wilcoxon Signed Rank Test Results for networks with four hidden layers, trained on CIFAR-100.
Each cell represents the results using a speciﬁc optimizer and regularizer, across all sparsity levels for that experiment setup as shown in ﬁgure 1. We use a p -value of 0.05, the bold values indicate where statistically sparse networks perform better than dense (reject Hfrom 4), while non-bold values indicate that it is possible dense networks networks have the same or better test accuracy in that conﬁguration (cannot reject H 0 from 4. The performance results for all these networks are present in Appendix C.
E MPIRICAL S ET - UP
SC-SDC MLP Setting We use the SC-SDC empirical setting (section 2.1) for all experiment variants.
We train over 6000 MLPs for 1000 epochs and evaluate performance on CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009).
We compare sparse and dense networks across various widths, depths, learning rates, regularization and optimization methods as shown in Table 2.
We choose a max network width N MaxW of n + 4 , where n is the input dimension of the network.
In the case of CIFAR, n = 3072 and so our maximum width N MaxW = 3076 .
We repeat these experiments with one, two and four hidden layers, with the number of active weights in these networks ranging from 949 , 256 to 31 , 765 , 568 weights.
In section 4, we discuss results achieved using four hidden layers and we provide the one and two hidden layer results in Appendix C.
Dense Width Following from SC-SDC , these networks are compared at various network widths, speciﬁcally a width of 308, 923, 1538, 2153, 2768 (10%, 30%, 50%, 70% and 90% of our maximum width N MaxW (3076)) as shown in Table 2.
We use the term dense width to refer to the width of a network if that network was dense.
For example, when comparing sparse and dense networks at a dense width of 308, this means the dense network has a width of 308, while the sparse network has a width of N MaxW (3076), but has the same number of active connections as the dense counterpart. We provide more detailed discussion of the choices made in the SC-SDC implementation in Appendix A.1.
Extended CNN Setting We also extend our experiments to Wide Resnet-50 (the WRN-28-10 variant) (Zagoruyko & Komodakis, 2016) and use the optimization and regularization conﬁgurations from the paper.
R ESULTS AND D ISCUSSION 4. C OMPARISON OF DENSE AND SPARSE INTERVENTIONS USING SC-SDC
In this section, we present the results of our analysis of sparse training dynamics (optimization, regularization and normalization).
We do this by using the results of the Wilcoxon signed rank test form SC-SDC to identify where optimization choices are currently well suited for sparse networks
We show the test accuracy (upper image) and gradient ﬂow (lower image) results for Sparse MLPs with four hidden layers.
These results compare the base version of a sparse network (no regularization or normalization), which is represented by the black line, with certain interventions such as adding regularization, normalization or changing the activation function.
These measures are displayed across varying density levels, with test accuracy being recorded at the end of training and gradient ﬂow averages over the training period.
For each line, the ﬁrst part of the label represents the networks regularization or normalization (No Reglarization, Data Augmentation, L 2 regularization and batch normalization), while the second part of the label references the activation function used (ReLU, SReLU and PReLU). and which are not. For each variant, we also measure the gradient ﬂow using EGF 1 as described in the previous section.
Our main ﬁndings are summarized as follows: Weight decay and data augmentation can hurt sparse network optimization, particularly when used with adaptive optimization methods such as Adagrad, Adam or RMSProp. 2.
Batch normalization is critical to training sparse networks, more so than it is for dense networks.
This suggests that gradient instability is a key obstacle for sparse optimization.
When training sparse networks using adaptive optimization methods, SReLU is a promising activation function.
On the other hand, when stochastic gradient is used (with and without momentum), PReLU performs the best across all the optimizers tested.
Weight Decay and Data Augmentation can hurt sparse network optimization From Table 3a, we can see that weight decay and data augmentation, in most conﬁgurations, does not favour sparse networks over dense networks.
When we take a closer look at the effects of these regularization methods on sparse network accuracy (Figure 2), we see that weight decay and data augmentation generally decrease accuracy when used with adaptive optimization methods, except for data augmentation which has an increase in performance when used with Adagrad.
In the case of weight decay, this decrease in accuracy is particularly apparent at high levels of sparsity (50% or less weights remaining), while for data augmentation this decrease is more drastic when Adam is used.
[Table] Batch normalization plays a disproportionate role in stabilizing sparse networks Batch normalization ensures that the distribution of the nonlinearity inputs remains stable as the network trains, which was hypothesized to help stabilize gradient propagation (gradients do not explode or vanish) (Ioffe & Szegedy, 2015).
Following from Table 3a, we see that batch normalization is statistically more important for sparse networks then dense networks, across most conﬁgurations.
From the test accuracy results presented in Figure 2, we also see that batch normalization improves performance across all sparse optimizers, while resulting in a lower, more stable gradient ﬂow.
This is especially apparent for adaptive gradient methods such as Adam and RMSProp.
Skip Connections are also useful for training sparse networks, with performance improvements across all optimizers, but this improvement is less drastic that the improvement achieved from using batch normalization.
Also, like batch normalization, skip connections result in a lower, more stable gradient ﬂow for all conﬁgurations, except when using stochastic gradient descent without momentum.
The potential of non-saturating activation functions - SReLU and PReLU We also explore the effect of different activation functions on sparse network optimization.
From Table 3b, we see that for Adam and RMSProp, most activation variants beneﬁt sparse networks more than dense networks.
For the other optimizers, different activation functions are not more favourable for sparse networks.
We take a closer look at two promising, non-saturating activation functions - SReLU (Jin et al., 2015) and PReLU (He et al., 2015).
From Figure 2, we see that when training sparse networks using adaptive optimization methods, SReLU is a promising activation function, performing better than all other optimizers.
Speciﬁcally when used with Adam and RMSProp (see 15 for activation functions),
SReLU has a similiar performance beneﬁt to that of using ReLU with batch normalization.
This could be due to the formulation of SReLU, which is made up of three piecewise functions, with no points of gradient saturation (unlike ReLU)
(Jin et al., 2015).
The increased gradient ﬂow could be the reason why SReLU has achieved promising results when used with sparse networks (Mocanu et al., 2018; Dubowski, 2020).
On the other hand, when stochastic gradient descent is used (with and without momentum), PReLU performs the best across all the optimizers tested (see 15 for activation functions).
4. G ENERALIZATION OF RESULTS ACROSS ARCHITECTURE TYPES .
In this section, we move on from SC-SDC and extend our result to WideResnet-50.
It is important to note that in Section 4.1, we tested the effect of interventions in isolation, while in this section we test the effect of certain intervention together.
We note from Table 4, that batch normalization remains critical to both dense and sparse network training.
At times (density of 20% and 100%), these networks fail to converge without it, while skip connections, even in deeper networks appear to be less critical.
Furthermore, we see the importance of L 2 and data augmentation, whose removal hurt both sparse and dense networks.
We also see that resampling the active weights (3), beneﬁts sparse networks.
R ELATED WORK
Pruning at Initialization Methods that prune at initialization aim to start sparse, instead of ﬁrst pre-training an overparameterized network and then pruning. These methods use certain criteria to estimate at initialization, which weights should remain active.
This criteria includes using the connection sensitivity (Lee et al., 2018b), gradient ﬂow (via the Hessian vector product) (Wang et al., 2020) and conversation of synaptic saliency (Tanaka et al., 2020).
Another branch of pruning is Dynamic Sparse Training, which uses information gathered during the training process, to dynamically update the sparsity pattern of these sparse networks (Mostafa & Wang, 2019; Bellec et al., 2017; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2019)
While our work is motivated by the same goal of allowing networks to start sparse and converge to the same performance as dense networks, we instead focus on the impact of optimization and regularization choices on sparse networks.
Sparse Network Optimization as Pruning Criteria Optimization in sparse networks has often been neglected in favour of studying network initialization.
However, there has been work that has looked at sparse network optimization from different perspectives, mainly as a guide for pruning criteria.
This includes using gradient information (Mozer & Smolensky, 1989; LeCun et al., 1989; Hassibi & Stork, 1992; Karnin, 1990), approximates of gradient ﬂow (Wang et al., 2020; Dettmers & Zettlemoyer, 2019;
Evci et al., 2020) and Neural Tangent Kernel (NTK) (Liu & Zenke, 2020) to guide the introduction of sparsity.
Sparse Network Optimization to study Network Dynamics Apart from use as pruning criteria, optimization information has been used to investigate aspects of sparse networks, such as their loss landscape (Evci et al., 2019), how they are impacted by SGD noise (Frankle et al., 2019a), the effect of different activation functions (Dubowski, 2020) and their weight initialization (Lee et al., 2019).
Our work differs from these approaches as we consider more aspects of the optimization and regularization process in a controlled experimental setting (SC-SDC), while using EGF to reason about some of the results.
C ONCLUSION AND F UTURE W ORK
In this work, we take a wider view of sparse optimization strategies and introduce appropriate tooling to measure the impact of architecture and optimization choices on sparse networks ( EGF , SC-SDC ).
Our results show that weight decay and data augmentation can hurt sparse network optimization, particularly when used in conjunction with adaptive optimization methods.
Furthermore, we show how batch normalization is critical to training sparse networks, more so than it is for dense networks as it helps stabilize gradient ﬂow.
Lastly, we also show the potential of non-saturating activation functions for sparse networks such as SReLU and PReLU.
A SC-SDC
In this section, we provide more information about SC-SDC and its beneﬁts. A. SC-SDC IMPLEMENTATION DETAILS
Wilcoxon Signed Rank Test
This is a non-parametric test that compares dependent or paired samples, without assuming the differences in between the paired experiments are normally distributed (McDonald, 2009; Demˇsar, 2006).
Random Sparsity Our work focuses on the training dynamics of random, sparse networks.
This ensures that what is learned is not dependent on a speciﬁc pruning method, but rather can be used to better understand sparse training in general.
Going forward, it would be interesting to explore these dynamics on pruned networks.
We achieve random sparsity, by generating a random mask for each layer and then multiply the weights by this mask during each forward pass.
The sparsity is distributed evenly across the network.
For example, a 20% sparse MLP has 20% of the weights remaining in each layer.
Dense Width A critical component to how we specify our experiments is a term we deﬁne as dense width.
In order to fairly compare sparse and dense networks, we need them to have the same number of active connections at each depth.
In the case of sparse networks, this means ensuring they have the same number of active connections as the dense networks, while remaining sparse.
Dense width refers to the width of a network if that network was dense. This process of comparing sparse and dense networks at different dense widths is illustrated in ﬁgure 4. Fair comparison of Sparse and Dense networks As can be seen from ﬁgure 4, SC-SDC ensures the exact same active parameter count, but the sparse networks will be connected to more neurons. It is possible that the increased number of activations being used can lead to sparse networks having higher representational power, however most work on expressivity of neural networks looks at this from a depth perspective and proves certain depths of networks are universal approximators (Eldan & Shamir, 2016;
Hornik et al., 1989; Funahashi, 1989).
To this end, we ensure these networks have the same depth, but we believe going forward an interesting direction would be ensuring they have a similar amount of active neurons.
SC-SDC comparison details For completeness, we provide more details of how we ensure sparse and dense networks are of the same capacity.
Following from equation 2, to ensure the same number of weights in sparse and dense networks, we can ensure they have the same number of active weights at each layer as follows:
[Equation]
This is achieved by masking each of the weight layers of sparse network S :
[Equation]
, where m l is a random binary matrix (mask) for layer l , s.t.
||
{+m l || 
+}
= a lD , where a lD is determined by the chosen capacity, these networks will be compared at.
For SC-SDC , we need a maximum network width N MaxW and comparison width N W .
We choose a max network width N MaxW of n + 4 , where n is the input dimension of the network.
In the case of CIFAR, n = 3072 and so our maximum width N MaxW = 3076 .
The choice of n + 4 follows from Lu et al.
(2017), where the authors prove a universal approximation theorem for width-bounded
ReLU networks, with width bounded to n + 4 .
Our comparison width, N W , is equivalent to dense widths we vary in our experiments - 308, 923, 1538, 2153, 2768.
The dimensions of each of layers are as follows:
First Layer:
[Equation]
Intermediate Layers
[Equation]
Final Layer
[Equation]
, where N MaxW is maximum width of the sparse layer, N W is the comparison width, I is the input dimension, O is output dimension, L is the number of layers in the network, θ lS is the weights in layer l of sparse network S and θ lD is the weights in layer l of dense network D .
This process would be the same for convolutional layers, but there would be a third dimension to handle the different channels.
In ﬁgure 3, we provide an illustrative example showing how to ensure sparse and dense networks are compared fairly.
A. B ENEFITS
The beneﬁts of SC-SDC can be summarized as follows:
• We can better understand sparse network optimization.
SC-SDC allows us to identify which optimization or regularization methods are poorly suited to sparse networks in a controlled setting, ensuring the results are a direct result of the sparse connections themselves.
Learn at what parameter and size budget, sparse networks are better than dense.
Comparing sparse and dense networks of the same capacity allows us to see which architecture is better at different conﬁgurations.
In conﬁgurations where sparse architectures perform better, we could exploit advances in sparse matrix computation and storage (Zhao et al., 2018; Merrill & Garland, 2016) to simply default to sparse architectures.
B G RADIENT F LOW
For completeness, we present the full set of results using the different formulations of gradient ﬂow on CIFAR-100.
Namely, we show ||
g
|| 1 (5) (Figure 5) ,
||
g || 2 (5) (Figure 6) , egf 1 (6) (Figure 7) and egf 2 (6) (Figure 8).
C D ETAILED R ESULTS FOR SC-SDC
In this section, we presented the detailed result for our experiments.
Detailed Results with a low learning rate (0.001). Detailed Results with a high learning rate (0.1). Results for different activation functions.
[Table] [Table] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Figure] [Table] [Table] [Figure]