The curse of (non)convexity: The case of an Optimization-Inspired Data Pruning algorithm
Abstract
Data pruning consists of identifying a subset of the training set that can be used for training instead of the full dataset.
This pruned dataset is often chosen to satisfy some desirable properties.
In this paper, we leverage some existing theory on importance sampling with Stochastic Gradient Descent (SGD) to derive a new principled data pruning algorithm based on Lipschitz properties of the loss landscape.
The goal is to identify a training subset that accelerates training (compared to random data pruning).
We call this algorithm LiPrune .
We illustrate cases where
LiPrune outperforms existing methods and discuss the limitations and failures of this algorithm in the context of deep learning.
Introduction
Data pruning aims at selecting a fraction of the most informative examples in a large training dataset.
This is done once, usually at initialization of after a few iterations, and the coreset remains unchanged thereafter.
It is an old topic that has amassed a large body of works [Welling, 2009, Chen et al., 2012,
Feldman et al., 2011, Huggins et al., 2016, Campbell and Broderick, 2019].
The main objective of data pruning is to reduce the computational complexity and achieve faster training with a lower power consumption.
Applications include: 1) neural architecture search (NAS) where models trained with a small fraction of the data serve as a proxy to quickly estimate the performance of a given choice of hyperparameters [Coleman et al., 2019]. 2)
Continual/incremental learning:
In the context of online learning, in order to avoid the forgetting problem, one keeps track of the most representative examples of past observations [Aljundi et al., 2019].
A variety of approaches can be used in order to select the relevant examples that will constitute the coreset.
Examples include Error based approaches where the goal is to find the most ‘difficult’ examples defined as the ones that contribute the most to the error: keeping the most foregettable examples (that change the most often from being well classified to being misclassified during the course of the training, Toneva et al.
[2018]), or the examples with highest expected gradient norm (GraND, EL2N scores Paul et al.
Another approach is the decision boundary based algorithms which find the examples near the decision boundary, the points for which the prediction has the highest variation (with respect to the input space, Ducoffe and
Precioso [2018], Margatina et al.
In this work, we focus on a different question: Can we prune the dataset in a way that accelerates training, as compared to e.g. random data pruning?
We answer this question by introducing a new data pruning algorithm, called LiPrune , based on the
Lipschitz coefficients of the loss function evaluated on single datapoints.
LiPrune is designed to accelerate the convergence of SGD in the context of convex optimization.
We document cases where
LiPrune succeeds at accelerating the training (convex case) and explain why this algorithm fails in the context of deep learning (non-convex case).
A comprehensive discussion is provided at the end of the paper.
SGD, Importance Sampling, and Stochastic data pruning
For some m ≥ 1 , consider an optimization problem of the form
[Equation]
where ( f i ) 1 ≤ i ≤ n is a sequence of functions from R m to R .
Given a dataset D = { ( x i , y i ) , i = 1 , . . .
, N } , we are particularly interested in two choices of the functions
f i : 1.
f i as point loss: in this case n = N and f i ( w ) =
ℓ ( y out ( x i ) , y i ) , where y out is a neural network and ℓ is some loss function, e.g. squared loss for regression, cross-entropy loss for classification.
f i as batch loss: in this case N = n × B where B ∈ N is the batch size and f i ( w )
=
1 B (cid:80) Bi
=1 ℓ ( y out ( x γ ( i ) ) , y γ ( i ) ) , and γ is a permutation of the set { 1 , . . . , N } .
This is the standard setting for neural networks training with SGD; batches are fixed at initialization.
When the number of functions f i ’s is large, computing the full gradient ∇ w F might be expensive.
Stochastic Gradient Descent (SGD) uses a randomly sampled index i from the set { 1 , . . . , n } to estimate the full gradient with the noisy version ∇ w f i 2 .
At training iteration t + 1 , SGD updates the weights according to the following rule
[Equation]
where i t is a randomly sampled index in { 1 , . . . , n } , and η is the learning rate.
The standard sampling distribution of i is the uniform distribution.
Let us assume that there exists a global minimum w ∗ = argmin w F ( w ) .
We have the following result on the convergence of SGD.
Theorem 1 (Thm 2.1 and Cor 2.2 in Needell et al.
[2014]) Assume that each f i is convex and that ∇ w f i has Lipschitz constant L i such that there exists a constant L sup with L i ≤ L sup almost surely.
Let F ( w ) =
E i [ f i ( w )]
= 1 n (cid:80) ni
=
f i ( w ) be µ − strongly convex and σ 2 =
E i [ ∥∇ w f i ( w ∗ ) ∥ 22 ] .
Given ϵ > 0 , and learning rate η = µϵ 2 µϵL sup +2 σ 2 , we have E ∥ w t − w ∗ ∥ 2 ≤ ϵ after t = 2 log(2 ϵ 0 /ϵ ) (cid:16) L sup [Equation] + [Equation]
[Equation] 2 (cid:17) , where ϵ [Equation]
Now assume that we sample the index i according to some probability vector p
= ( p 1 , p 2 , . .
. , p n ) such that p T 1 = (cid:80)
ni =1 p i = 1 and p i ∈ (0 , 1) .
The new SGD update rule is given by
[Equation]
where g i ( w ) = 1 np i f i ( w ) .
The normalization by n p i is necessary so that we have an unbiased estimate of the gradient, i.e. E [ ∇ w g
i ( w )]
= ∇ F ( w ) .
This is equivalent to the minimization problem Eq.
(1) when f i is replaced by g i .
Assuming that ∇ w f i has Lipschitz constant L i , the function g i has Lipschitz constant L
i np i .
Therefore, the new L sup (which depends on p ) is given by
L sup ( p ) =
sup i
L
i np i .
From Theorem 1, the convergence rate is given by 1 − 2 ηµ (1 − ηL sup ( p )) , and a smaller L sup ( p ) results in faster convergence.
Hence, a natural choice of the probability vector p would be the one that minimizes L sup ( p ) , i.e. p ∗ =
argmin p
L sup ( p ) under the condition that p T 1 = 1 .
This has a closed-form solution given by p ∗
i
=
L i ¯ L where ¯ L = 1 n (cid:80) ni
=1 L i (Needell et al.
Hence, choosing probabilities p i ∝ L i is optimal for convergence speed with importance sampling.
How do we leverage this result to perform data pruning?
Interpretation.
The Lipschitz constant L i encodes some information about the second order geometry of the function f i .
Assuming that f i has a second derivative, and that w ∈ C ⊂ R m where
C is a compact set, L i represents the maximum norm eigenvalue of the Hessian matrix of f i over the compact set C .
Thus, L i captures the maximum curvature of the function f
i over the set C .
Hence, sampling i according to p i ∝ L i entails giving more importance to indices i for which f
i has large curvatures.
We can think of these indices as those corresponding to hard functions f i .
For instance, in the context of ordinary SGD with batch size 1 where f i ( w ) =
ℓ ( y out ( x i ) , y i ) for ( x i , y i ) ∈ D , the optimal sampling scheme gives more importance to hard examples in the training procedure.
For mini-batch SGD, this corresponds to giving more importance to hard batches.
Stochastic pruning.
Now that we understand importance sampling in SGD, how do we incorporate data pruning in this framework?
Intuitively, given some compression ratio r ∈ (0 , 1) 3 , we can define a stochastic variant of data pruning as performing the following SGD updates
[Equation]
where ζ t ∼ B ( r ) is a Bernoulli variable with probability r , i.e. P ( ζ t = 1) = 1 − P ( ζ t = 0) =
r , i t is sampled according to some probability vector p and g i ( w ) def = 1 rnp i f i ( w ) .
With this update rule, when we train for n iterations, only a fraction r of the data will be sampled on average.
This can be seen a stochastic data pruning algorithm.
For this algorithm, we have the following convergence result, which is similar in flavour to that of Theorem 1.
Theorem 2 (SGD Convergence with Stochastic Data Pruning) Assume that each f i is convex and that ∇ w f i has Lipschitz constant L i such that there exists a constant L sup with L i ≤ L sup .
Let
F ( w ) =
E i [ f i ( w )] be µ − strongly convex and σ 2 =
E i [ ∥∇ w f i ( w ∗ ) ∥ 22 ] .
Consider the update rule given by Eq.
(2) , where g i ( w ) = 1 rnp i f i ( w ) , r ∈ (0 , 1) is the compression ratio, and i t is sampled according to some probability vector p .
Suppose η < r/L sup ( p ) where L sup ( p ) def = sup i L i np i , then with probability at least 1 − e − r 2 t 2 over the distribution of ζ = ( ζ 1 , . .
. , ζ t ) , we have that
[Equation]
p [Equation] where σ 2 ( p ) = 1 n 2 (cid:80) ni =1 1 p i ∥∇ w f i ( w ∗ ) ∥ 22 .
As a result, given some error threshold ϵ > 0 , by choosing η = µrϵ 2 σ 2 ( p )+2 µL sup ( p ) ϵ , with probability at least 1 − e − r 2 t 2 we have that E ∥ w t − w ∗ ∥ 2 ≤ ϵ for t = 4 log (cid:18) 2 ϵϵ (cid:19)
[Equation]
r
[Equation]
(cid:19) .
The first part of the proof of Theorem 2 is similar to that of Theorem 2.1 in Needell et al.
[2016] and uses the same co-coercivity property.
The second part is an application of Hoeffding’s inequality to control the deviation of (cid:80) k ≤ t ζ k from its mean.
An important consequence of Theorem 2 is that the convergence rate of the stochastic pruning algorithm Eq.
(2) is controlled by the term λ = 1 − 2 µ ηr (cid:0) 1 − ηr L sup ( p ) (cid:1) ; the smaller the quantity λ , the faster the convergence.
In the case of r = 1 , Needell et al.
[2016] characterizes the choice of the probability vector p that minimizes the growth factor λ , which is given by p i ∝ L i .
It is straightforward to see that this choice of p is optimal for any choice of compression ratio r .
Conclusion: To maximize the factor λ , the probability vector p should be chosen as p i =
[Equation]
j [Equation]
Theorem 2 shows that sampling according to L i ’s results in faster convergence of the stochastic data pruning algorithm given by Eq.
However, our goal in data pruning is to reduce the size of the dataset (deterministic pruning) and the stochastic pruning algorithm
(Eq.
(2)) does not fit in this category (each index i has a non-zero sampling probability).
Can we derive a detemrministic version of algorithm Eq.
(2) with the optimal choice of p ?
As straightforward deterministic variant is obtained by using the Lipschitz coefficients L i ’s as pruning scores.
This consists of keeping the samples i with the largest L i ’s.
We define this algorithm in the next section.
Deterministic pruning
The deterministic alternative of stochastic data pruning (Eq.
(2)) is to rank the functions f i based on the Lipschitz constants L i since it represents how hard the function f
i is to train.
We call this algorithm LiPrune , and we describe it in Algorithm 1.
LiPrune requires access to the Lipschitz coefficients L ′ i s , which are generally unknown.
However, approximations of L i ’s can be obtained in different ways: The compression ratio is the fraction of datapoints kept after pruning. • Using the Gradient.
Paul et al.
[2021] introduced the GraNd algorithm which uses the gradient norm to rank samples.
Let D =
( x i , y i ) , i = 1 , . . . , n } be a dataset and f i ( w ) =
ℓ ( y out ( x i , w ) , y i ) , where ℓ is some loss function and y out ( x i , w ) is the network output, and w are the weights of the network.
GraNd then computes approximations of the gradient norm given by χ i =
E w [ ∥∇ w f i ∥ ] where the expectation is taken over random initializations, and uses the scores ( χ i ) 1 ≤ i ≤ n as a measure of sample importance.
This gradient computation is usually performed at a fixed iteration t , e.g. pruning at initialization where t = 0 , or during training where t > 0 .
Recall that LiPrune uses the Lipschitz constant L i ’s as sample importance scores.
This Lipschitz constant satisfies |∇ w f i ( w ) − ∇ w f i ( w ∗ ) | ≤ L i ∥ w − w ∗ ∥ where w ∗ = argmin w F ( w ) .
Hence, given some weight w , we can use the fraction |∇ w f i ( w ) − ∇ w f i ( w ∗ ) | / ∥ w − w ∗ ∥ as an approximation of L i .
Assuming that ∇ w f i ( w ∗ ) ≈ [-
-]0 for all i , GraNd can be seen as an approximate LiPrune .
• Using the Hessian.
A better estimate of the Lipschitz constants L
i ’s is the norm of the Hessian matrix.
Indeed, locally, we have that |∇ w f i ( w ′ ) − ∇ w f i ( w ) | ≤ ∥∇ 2 f i ( w ) ∥∥ w ′ − w ∥ , and the constant ∥∇ 2 f i ( w ) ∥ is tight.
We propose to use the Hessian matrix norm to estimate the Lipschitz constants L i ’s.
However, a major drawback of this methods is the computation cost of the Hessian matrix norm.
This cannot be performed for each example ( x i , y i ) in the dataset.
Hence, we restrict our analysis to batch pruning, and consider the second scenario where f i ( w ) =
B (cid:80) x i ∈ B i ℓ ( y out ( x i ) , y i ) .
Intuitively, using LiPrune in batch pruning should accelerate SGD convergence as compared to random batch pruning.
We summarize this in the following hypothesis.
Hypothesis.
By using the Hessian norm to estimate the Lipschitz constants L i ’s in batch pruning, we can accelerate the convergence speed with minimal loss in performance. Algorithm
Require:
Functions ( f i ) 1 ≤ i ≤ n with Lipschitz constants L i ’s, compression ratio r ∈ (0 , 1) .
s i ← L i (scores) ˆ
s i ← sorted ( s ) i ( i th largest score) T r ← ˆ s r × n (threshold) I = ∅ (to be populated with indices) for i = 1 , . . . , n do if s i ≥ T r then
I =
I ∪ { i } end if end for return
I
In the next section, we empirically verify our hypothesis by pruning and training models until convergence.
We observe that while the loss in performance might be indeed minimal, the convergence speed remains relatively unchanged.
We conjecture that this is due to the non-convex nature of the loss landscape, and explain why LiPrune might not be the best pruning algorithm in this case.
Experiments 4.1 Linear model
We consider a linear model where the data is generated using the following rule ( i ∈ [ N ] , N = 1000 )
[Equation]
(3) where w ∗ ∈ R is fixed and x i ∼ U ([ − 5 , 5]) (uniform distribution).
See Fig.
4 for an illustration of the dataset.
We would like to fit a linear regression model to the data D = { ( x i , y i ) , i = 1 , . . .
, N } by minimizing the mean squared loss.
Is this case, the functions f i are given by f i ( w ) = 12 ( w x i − y i ) 2 , and the Lipschitz constants are given by L i =
| x i | 2 .
Using LiPrune in this case implies choosing the samples ( x i , y i ) with the largest | x i
| 2 .
To understand why this would accelerate convergence, let us look at what happens with SGD.
In this case
L o ss r =0.
0 5 10 15 20 25 3013 r =0.
0 5 10 15 20 25 30 SGD steps 0.0 0.
L o ss r =0.
0 5 10 15 20 25 30 SGD steps 0.0 0.
r =0.
No pruning
LiPruneRandom pruning the update rule is given by w k +1 =
w k −
[Figure]
[Equation]
and hence the convergence speed is controlled by the contraction factor (1 − ηx 2 i k ) .
Choosing x i k such that x
2 i k is large results in faster convergence (on average).
Fig.
1 shows the empirical results of data pruning using LiPrune vs. Random pruning for different compression ratios r .
Notice that the initial loss value is large with LiPrune since we fix the initial point of SGD to w 0 = 0 .
The results confirm our theoretical predictions that LiPrune accelerates convergence.
More experiments with different hyper-parameters are provided in Appendix B.
4.2 Non-convex case: Neural Networks
To evaluate LiPrune on deep learning applications, we consider a ResNet architecture and the
CIFAR10 dataset.
We use batch pruning (based on the estimation of the Hessian norm) at different training iterations (pruning at initialization or later in training).
We train the model with SGD with batch size 128 and varying learning rates for 164 epochs.
In Fig.
2, we report the test error corresponding to using LiPrune with different network depths, compression ratios, and pruning iterations.
The learning rate in this case is 0 .
01 which is divided by 10 at epoch 80 and 120.
Experiments with different learning rates are provided in Appendix B.5.
From these empirical results, we conclude that there is no (training acceleration) advantage of using LiPrune (in the current form of batch pruning) instead of random pruning.
[Figure]
Discussion
The gains of the data pruning methods are usually less remarkable in deep learning.
In particular, they lead to limited improvement compared to naively prioritizing the samples uniformly at random.
For example, Guo et al.
[2022] conduct extensive experiments with 12 of the most popular data pruning methods on Cifar10 and imagenet datasets with popular architectures (ResNet-18, VGG-16,
Inception-v3) for variaous pruning levels, keeping from 0 .
1% to 100% of the data.
One of the main conclusions is that random pruning is a strong baseline: no method can outperform it systematically across the different levels of pruning;
Random pruning achieves the best test accuracy when pruning 90% or 70% of ImageNet data, for example.
As seen in the experiment section, our method suffers from the same caveat.
We believe that two different and potentially concurrent effects can explain these behaviors: i) modification of the asymptotic loss and ii) modification of the learning dynamic
Modification of the asymptotic loss.
In essence, any pruning algorithm aims to find a better subset than random pruning; therefore, it transforms the data distribution, i.e. the distribution of ( X, Y ) .
This typically results in a modification of the loss landscape.
We illustrate this point using the example developed in Appendix B.4.
The data generating process is given by
[Equation]
where w ∗ = 12 , and y out ( x ; w ) = sin( wx ) .
In Figure 3, we compare the loss landscape that is minimized without pruning (left figure), when keeping only 50% , and 1% of the data (middle and right figures) with LiPrune.
For r = 0 .
5 , we can still ’easily’ identify the minimizer w ∗ .
With r=0.01, the task is more challenging.
Though not always mentioned, we believe this behavior is common to many popular pruning algorithms, if not most.
For example, the authors of Kawaguchi and Lu [2020], are able to derive a bound on the loss transformation induced by their pruning algorithm in the slightly different setting of dynamic pruning.
The authors prove that their algorithm asymptotically returns a global minimum for convex losses.
However, there are no such results for general non-convex losses.
[Figure]
Modification of the learning dynamic.
In the convex setting, it is sufficient to reduce the stochastic gradient variance to converge faster, which is the base ingredient of accelerated SGD with importance sampling.
In the highly multimodal landscapes of neural network problems, the noise of the gradient plays a pivotal role in the training dynamic and the final performance of the model.
Several lines of work argue, for instance, that stochastic procedures such as dropout or stochastic depth improve the generalization results because they introduce additional noise that acts as an implicit regularizer [Wei et al., 2020, Hayou and Ayed, 2021], smoothing the loss landscape.
It has long been believed that the noise in SGD favors flat minima [Heskes and Kappen, 1993].
In Xie et al.
[2020], the authors adopt a diffusion theory perspective to study the training dynamics and prove, under certain simplifying assumptions, that the noise in SGD exponentially favors flat minima.
A different line of work [Ma et al., 2018, Zhang et al., 2019] observes that SGD has two regimes, the noise dominated (small batches/large learning rate), characterized by the gradient noise, and curvature dominated (large batches/small learning rate).
In Smith et al.
[2020], the authors empirically verify the existence of the two regimes and confirm that the small batch size (larger noise) leads to test accuracy improvements when compared to large batch sizes (smaller noise).
The noise is particularly determinant in the early phase of the training.
We argue that a successful data pruning method should consider the changes it induces in the training dynamics and the noise of the gradient.
In Johnson and Guestrin [2018], for example, a key component of the method is to adapt the learning rate by rescaling it by the inverse of the estimated gradient noise reduction.
A Proof of Theorem
Theorem 2 (SGD Convergence with Stochastic Data Pruning).
Assume that each f i is convex and that ∇ w f i has Lipschitz constant L i such that there exists a constant L sup with L i ≤ L sup almost surely.
Let F ( w ) =
E i [ f i ( w )] be µ − strongly convex and σ 2 =
E i [ ∥∇ w f i ( w ∗ ) ∥ 22 ] .
Consider the following SGD update
[Equation]
where g i ( w ) = 1 rnp i f i ( w ) , r ∈ (0 , 1) is the compression ratio, and i is sampled according to some probability vector p .
Suppose η < r/L sup ( p ) where L sup ( p ) def = sup i L i np i , then with probability at least 1 − e − r 2 t 2 over the distribution of ζ = ( ζ 1 , . .
. , ζ t ) , we have that
[Equation]
, where σ 2 ( p ) = 1 n 2 (cid:80) ni =1 1 p i ∥∇ w f i ( w ∗ ) ∥ 22 .
As a result, given some error threshold ϵ > 0 , by choosing η = µrϵ 2 σ 2 ( p )+2 µL sup ( p ) ϵ , with probability at least 1 − e − r 2 t 2 we have that E ∥ w t − w ∗ ∥ 2 ≤ ϵ for t = 4 log (cid:18) 2 ϵϵ
(cid:19)r (cid:18)
L
[Equation]
( p ) µ +
[Equation]
( p ) µ
[Equation]
ϵ (cid:19) .
The first part of the proof of Theorem 2 is similar to that of Theorem 2.1 in Needell et al.
[2016] in the sense that it uses the same co-coercivity property.
The second part is an application of Hoeffding’s inequality to control the deviation of (cid:80) k ≤ t ζ k from its mean.
Lemma 1 (Co-coercivity, Lemma A.1 in Needell et al.
Let f be a smooth function with
Lipschitz constant L , then for all x, y
[Equation]
Now let η k def =
ηζ k .
We have that
[Equation]
Jensen’s inequality)
[Equation]
(Co
[Equation]
Taking the expectation with respect to i t , and letting σ 2 ( p )
= 1 n 2 (cid:80) ni
=1 1 p i ∥∇ w f i ( w ∗ ) ∥ 22 , we obtain
[Equation]
Using strong convexity of F we have
[Equation]
Let λ = 1 − 2 µ ηr (cid:0) 1 − ηr L sup ( p ) (cid:1) and χ t = (cid:80) tj =1 ζ j .
Using this recursively [Equation] we obtain for all t
[Equation]
Lemma 2 (Hoeffding’s inequality) Let X 1 , . . .
, X n be independent random variables such that a i ≤ X i
≤ b i almost surely.
Let S n = X 1 + · · · + X n , then for all z > 0 , we have that
[Equation]
Using Hoefdding’s inequality (Lemma 2) with z = rt/ 2 , we have that with probability at least 1 − 2 exp( − r 2 t 2 ) χ
[Equation]
[Equation]
= rt/ 2 . Hence, with probability at least 1 − 2 exp( − r 2 t 2 ) , we have that
[Equation]
6) which concludes the first part of the proof.
Now let ϵ > 0 and consider the following choice of the learning rate η
[Equation]
With this choice, we have that
[Equation]
Now it suffices to control the first term of Eq.
(6) to conclude.
Let ϵ 0 =
∥ w 0 − w ∗ ∥ 22 .
In order to to have λ rt/ 2 ϵ 0 ≤ ϵ/ 2 , it is sufficient to have
[Equation]
As in Needell et al.
[2016], we leverage the inequality − 1 log(1 − x ) ≤ 1 /x with x = 1 − λ to obtain a sufficient condition for this statement to hold.
Indeed, using this inequality, it is straightforward that rt 2 × log ( λ ) ≤ log( ϵ/ 2 ϵ 0 ) is satisfied whenever t ≥ 4 log (cid:18) 2 ϵϵ
(cid:19)r (cid:18)
L sup ( p ) µ + σ 2 ( p ) µ 2 ϵ (cid:19) .
B Additional experiments B.1 Data distribution in the linear model
[Figure]
[Equation]
[Figure]
B.2 Different learning rates
We show in Fig.
5 and Fig.
6 the results of LiPrune for different compression ratios with a different learning rates.
The results show consistent training acceleration with LiPrune , compared to random pruning.
In Fig.
6, the convergence is fast for both LiPrune and random pruning, with a small advantage for LiPrune .
[Figure] [Figure]
In Fig.
[Figure]
Fig.
[Figure]
Fig.
[Figure]
we show the empirical results of LiPrune (Vs random pruning) for different choices of the batch size.
LiPrune seems to be robust to batch size as well, in the case of the linear model.
As the batch size increases (Fig. 9), the empirical advantage of LiPrune over random pruning becomes less significant, which is expected since the pruned sets obtained using LiPrune and random pruning have more and more shared samples.
[Figure] [Figure]
We consider a simple non-linear model where the data is generated using the following rule (for i ∈ [ N ] , N = 1000 )
[Equation]
where w ∗ = 12 is fixed and x i ∼ N (0 , 1) (standard normal distribution).
See Fig.
10 for an illustration of the dataset.
We would like to fit a regression model of the form y = sin( wx ) to the data D = { ( x i , y i ) , i = 1 , . . .
, N } by minimizing the mean squared loss.
We show in Fig.
[Figure]
[Figure]
=
[Figure] r =0. [Figure] the empirical loss landscape with N = 1000 samples.
The figure illustrates, as expected, a highly non-convex shape.
[Figure]
[Equation]
in Fig.
12, we show the training curves obtained with LiPrune and random pruning.
In this case, there is no clear (acceleration) advantage of LiPrune .
B.5 ResNet experiments
In Figures 13 and ??
we report the results when pruning respectively at iteration 1 and 5 for different pruning rates r when the learning rate is η = 0 .
Figures 15 and ??
show the same plots with learning rate η = 0 . We can see that the conclusions are identical. We also performed the same experiments when pruning at iteration 15 and obtained very similar plots. [Figure]
ss
[Figure]
LiPrune
[Figure] [Figure]
[Figure] [Figure]
[Figure]