Original: O N THE P REDICTABILITY OF P RUNING A CROSS S CALES
Revised: O N THE P REDICTABILITY OF P RUNING A CROSS S CALES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We show that the error of iteratively-pruned networks  follows a scalinglaw, and that this law is of a fundamentally different nature than that of unpruned networks.
Revised: We show that the error of iteratively-pruned networks empirically follows a scaling law with interpretable coefﬁcients that depend on the architecture and task.

Original: We functionally approximate the error of the pruned networks, showing that it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different sparsities are freely interchangeable.
Revised: We functionally approximate the error of the pruned networks, showing that it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different sparsities are freely interchangeable.

Original: We demonstrate the accuracy of this functional approximation over scales spanning orders of magnitude in depth, width, dataset size, and sparsity for CIFAR-10 and ImageNet; and show that it generalizes over different architectures  and  pruning algorithms.
Revised: We demonstrate the accuracy of this functional approximation over scales spanning orders of magnitude in depth, width, dataset size, and sparsity. We show that the scaling law functional form holds (generalizes) for large scale data (CIFAR-10, ImageNet), architectures (ResNets, VGGs) and iterative pruning algorithms (IMP, SynFlow).

Original: As neural networks become ever larger and more expensive to train, our ﬁndings enable a framework for reasoning conceptually and analytically about pruning.
Revised: As neural networks become ever larger and more expensive to train, our ﬁndings suggest a framework for reasoning conceptually and analytically about pruning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For decades, neural network pruning —eliminating unwanted parts of the network—has been a popular approach for reducing network sizes or computational demands of inference (LeCun et al., 1990;
Revised: For decades, neural network pruning —eliminating unwanted parts of the network—has been a popular approach for reducing network sizes or computational demands of inference (LeCun et al., 1990;

Original: Reed, 1993; Han et al., 2015).
Revised: Reed, 1993; Han et al., 2015).

Original: In practice, pruning can reduce the parameter-counts of contemporary models by 2x (Gordon et al., 2020) to 5x (Renda et al., 2020) with no reduction in accuracy.
Revised: In practice, pruning can reduce the parameter-counts of contemporary models by 2x (Gordon et al., 2020) to 5x (Renda et al., 2020) with no reduction in accuracy.

Original: More than 80 pruning techniques have been published in the past decade (Blalock et al., 2020), but, despite this enormous volume of research, there remains little guidance on important aspects of pruning.
Revised: More than 80 pruning techniques have been published in the past decade (Blalock et al., 2020), but, despite this enormous volume of research, there remains little guidance on important aspects of pruning.

Original: Consider a seemingly simple question one might ask when using a particular pruning technique:
Revised: Consider a seemingly simple question one might ask when using a particular pruning technique:

Original: Given a family of neural networks (e.g., ResNets on ImageNet of various widths and depths), which family member should we prune (and by how much) to obtain the network with the smallest parametercount such that error does not exceed some threshold (cid:15) k ?
Revised: Given a family of neural networks (e.g., ResNets on ImageNet of various widths and depths), which family member should we prune (and by how much) to obtain the network with the smallest parametercount such that error does not exceed some threshold (cid:15) k ?

Original: As a ﬁrst try, we could attempt to answer this question using brute force: we could prune every member of a family (i.e., perform grid search over widths and depths) and select the smallest pruned network that satisﬁes our constraint on error.
Revised: As a ﬁrst try, we could attempt to answer this question using brute force: we could prune every member of a family (i.e., perform grid search over widths and depths) and select the smallest pruned network that satisﬁes our constraint on error.

Original: However, depending on the technique, pruning one network (let alone grid searching) could take days or weeks on expensive hardware.
Revised: However, depending on the technique, pruning one network (let alone grid searching) could take days or weeks on expensive hardware.

Original: If we want a more efﬁcient alternative, we will need to make assumptions about pruned networks: namely, that there is some structure to the way that their error behaves.
Revised: If we want a more efﬁcient alternative, we will need to make assumptions about pruned networks: namely, that there is some structure to the way that their error behaves.

Original: For example, that pruning a particular network changes the error in a predictable way.
Revised: For example, that pruning a particular network changes the error in a predictable way.

Original: Or that changing the width or depth of a network changes the error when pruning it in a predictable way.
Revised: Or that changing the width or depth of a network changes the error when pruning it in a predictable way.

Original: We could then train a smaller number of networks, characterize this structure, and estimate the answer to our question.
Revised: We could then train a smaller number of networks, characterize this structure, and estimate the answer to our question.

Original: We have reason to believe that such structure does exist for pruning: techniques already take advantage of it implicitly.
Revised: We have reason to believe that such structure does exist for pruning: techniques already take advantage of it implicitly.

Original: For example, Cai et al.
Revised: For example, Cai et al.

Original: (2019) create a single neural network architecture that can be scaled down to many different sizes; to choose which subnetwork to deploy, Cai et al.
Revised: (2019) create a single neural network architecture that can be scaled down to many different sizes; to choose which subnetwork to deploy, Cai et al.

Original: train an auxiliary, black-box neural network to predict subnetwork performance.
Revised: train an auxiliary, black-box neural network to predict subnetwork performance.

Original: Although this black-box approach implies the existence of structure, it does not reveal this structure explicitly or make it possible to reason analytically in a fashion that could answer our research question.
Revised: Although this black-box approach implies the existence of structure, it does not reveal this structure explicitly or make it possible to reason analytically in a fashion that could answer our research question.

Original: Outside the context of pruning algorithms, such structure has been observed—and further codiﬁed explicitly—yielding insights and predictions in the form of scaling laws.
Revised: Outside the context of pruning algorithms, such structure has been observed—and further codiﬁed explicitly—yielding insights and predictions in the form of scaling laws.

Original: Tan and Le (2019) design the EfﬁcientNet family by developing a heuristic for predicting efﬁcient tradeoffs between depth, width, and resolution.
Revised: Tan and Le (2019) design the EfﬁcientNet family by developing a heuristic for predicting efﬁcient tradeoffs between depth, width, and resolution.

Original: Hestness et al.
Revised: Hestness et al.

Original: (2017) observe a power-law relationship between dataset size and the error of vision and NLP models.
Revised: (2017) observe a power-law relationship between dataset size and the error of vision and NLP models.

Original: Rosenfeld et al.
Revised: Rosenfeld et al.

Original: (2020) use a power scaling law to predict the error of all variations of architecture families and dataset sizes jointly, for computer vision and natural language processing settings.
Revised: (2020) use a power scaling law to predict the error of all variations of architecture families and dataset sizes jointly, for computer vision and natural language processing settings.

Original: Kaplan et al.
Revised: Kaplan et al.

Original: (2020) develop a similar power law for language models that incorporates the computational cost of training.
Revised: (2020) develop a similar power law for language models that incorporates the computational cost of training.

Original: Inspired by this line of work, we address our research question about pruning by developing a scaling law to predict the error of networks as they are pruned.
Revised: Inspired by this line of work, we address our research question about pruning by developing a scaling law to predict the error of networks as they are pruned.

Original: To the best of our knowledge, no explicit scaling law holding over pruning algorithms and network types currently exists.
Revised: To the best of our knowledge, no explicit scaling law holding over pruning algorithms and network types currently exists.

Original: In order to formulate such a predictive scaling law, we consider the dependence of generalization error on the pruning-induced density for networks of different depths and width trained on different dataset sizes.
Revised: In order to formulate such a predictive scaling law, we consider the dependence of generalization error on the pruning-induced density for networks of different depths and width trained on different dataset sizes.

Original: We begin by developing a functional form that accurately estimates the generalization error of a speciﬁc model as it is pruned (Section 3).
Revised: We begin by developing a functional form that accurately estimates the generalization error of a speciﬁc model as it is pruned (Section 3).

Original: We then account for other architectural degrees of freedom, expanding the functional form for pruning into a scaling law that jointly considers density alongside width, depth, and dataset size (Section 4).
Revised: We then account for other architectural degrees of freedom, expanding the functional form for pruning into a scaling law that jointly considers density alongside width, depth, and dataset size (Section 4).

Original: The basis for this joint scaling law is an invariant we uncover that describes ways that we can interchange depth, width, and pruning without affecting error.
Revised: The basis for this joint scaling law is an invariant we uncover that describes ways that we can interchange depth, width, and pruning without affecting error.

Original: The result is a scaling law that accurately predicts the performance of pruned networks across scales.
Revised: The result is a scaling law that accurately predicts the performance of pruned networks across scales.

Original: Finally, we use this scaling law to answer our motivating question (Section 6).
Revised: Finally, we use this scaling law to answer our motivating question (Section 7).

Original: The same functional form can accurately estimate the error for both unstructured magnitude pruning (Renda et al., 2020) and SynFlow (Tanaka et al., 2020) when ﬁt to the corresponding data, suggesting we have uncovered structure that may be applicable to iterative pruning more generally.
Revised: The same functional form can accurately estimate the error for both unstructured magnitude pruning (Renda et al., 2020) and SynFlow (Tanaka et al., 2020) when ﬁt to the corresponding data, suggesting we have uncovered structure that may be applicable to iterative pruning more generally.

Original: And now that we have established this functional form, ﬁtting it requires only a small amount of data (Appendix F).
Revised: And now that we have established this functional form, ﬁtting it requires only a small amount of data (Appendix 5).

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We develop a scaling law that accurately estimates the error when pruning a single network.
Revised: • We develop a scaling law that accurately estimates the error when pruning a single network.

Original: • We observe and characterize an invariant that allows error-preserving interchangeability among depth, width, and pruning density.
Revised: • We observe and characterize an invariant that allows error-preserving interchangeability among depth, width, and pruning density.

Original: • Using this invariant, we extend our single-network scaling law into a joint scaling law that predicts the error of all members of a network family at all dataset sizes and all pruning densities.
Revised: • Using this invariant, we extend our single-network scaling law into a joint scaling law that predicts the error of all members of a network family at all dataset sizes and all pruning densities.

Original: • In doing so, we demonstrate that there is structure to the behavior of the error of iteratively pruned networks that we can capture explicitly with a simple functional form.
Revised: • In doing so, we demonstrate that there is structure to the behavior of the error of iteratively pruned networks that we can capture explicitly with a simple functional form.

Original: • Our scaling law enables a framework for reasoning analytically about pruning, allowing us to answer our motivating question and similar questions about pruning.
Revised: • Our scaling law enables a framework for reasoning analytically about pruning, allowing us to answer our motivating question and similar questions about pruning.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Pruning.
Revised: Pruning.

Original: We study two techniques for pruning neural networks: iterative magnitude pruning (IMP) (Janowsky, 1989; Han et al., 2015; Frankle et al., 2020) in the main body of the paper and SynFlow (Tanaka et al., 2020) in
Revised: We study two techniques for pruning neural networks: iterative magnitude pruning (IMP) (Janowsky, 1989; Han et al., 2015; Frankle et al., 2020) in the main body of the paper and SynFlow (Tanaka et al., 2020) in

Original: Appendix E.
Revised: Appendix E.

Original: We describe IMP in detail here and SynFlow in Appendix A.
Revised: We describe IMP in detail here and SynFlow in Appendix A.

Original: IMP prunes by removing a fraction—typically 20%, as we do here—of individual weights with the lowest magnitudes at the end of training.
Revised: IMP prunes by removing a fraction—typically 20%, as we do here—of individual weights with the lowest magnitudes at the end of training.

Original: 1 We choose these weights globally throughout the network, i.e., without regard to speciﬁc layers.
Revised: 1 We choose these weights globally throughout the network, i.e., without regard to speciﬁc layers.

Original: We use per-weight magnitude pruning because it is generic, well-studied (Han et al., 2015), and matches the sparsity/accuracy tradeoffs of more complicated methods (Gale et al., 2019;
Revised: We use per-weight magnitude pruning because it is generic, well-studied (Han et al., 2015), and matches the sparsity/accuracy tradeoffs of more complicated methods (Gale et al., 2019;

Original: Blalock et al., 2020; Renda et al., 2020).
Revised: Blalock et al., 2020; Renda et al., 2020).

Original: Pruning weights typically reduces the accuracy of the trained network, so it is standard practice to further train after pruning to recover accuracy.
Revised: Pruning weights typically reduces the accuracy of the trained network, so it is standard practice to further train after pruning to recover accuracy.

Original: For IMP, we use a practice called weight rewinding , in which the values of unpruned weights are rewound to their values at epoch 10 and the training process is repeated from there to completion.
Revised: For IMP, we use a practice called weight rewinding , in which the values of unpruned weights are rewound to their values at epoch 10 and the training process is repeated from there to completion.

Original: To achieve sparsity levels beyond 20%, this process is repeated iteratively —pruning by 20%, rewinding, and retraining—until a desired sparsity level is reached.
Revised: To achieve sparsity levels beyond 20%, this process is repeated iteratively —pruning by 20%, rewinding, and retraining—until a desired sparsity level is reached.

Original: Renda et al.
Revised: Renda et al.

Original: (2020) demonstrate that IMP with weight rewinding achieves state-of-the-art tradeoffs between sparsity and accuracy.
Revised: (2020) demonstrate that IMP with weight rewinding achieves state-of-the-art tradeoffs between sparsity and accuracy.

Original: For a formal statement of this pruning algorithm, see Appendix A.
Revised: For a formal statement of this pruning algorithm, see Appendix A.

Original: Datasets.
Revised: Datasets.

Original: We study the image classiﬁcation tasks CIFAR-10 and ImageNet.
Revised: We study the image classiﬁcation tasks CIFAR-10 and ImageNet.

Original: Our scaling law predicts the error when training with the entire dataset and smaller subsamples .
Revised: Our scaling law predicts the error when training with the entire dataset and smaller subsamples .

Original: To subsample a dataset to a size of n , we randomly select n of the training examples without regard to individual classes such that in expectation we preserve the original dataset distribution (we always retain the entire test set).
Revised: To subsample a dataset to a size of n , we randomly select n of the training examples without regard to individual classes such that in expectation we preserve the original dataset distribution (we always retain the entire test set).

Original: When performing iterative pruning, we maintain the same subsample for all pruning iterations.
Revised: When performing iterative pruning, we maintain the same subsample for all pruning iterations.

Original: Networks.
Revised: Networks.

Original: We study three families of neural networks: ResNets for CIFAR-10, ResNets for ImageNet, and (in Appendix E) VGG-style networks for CIFAR-10.
Revised: We study three families of neural networks: ResNets for CIFAR-10, ResNets for ImageNet, and (in Appendix E) VGG-style networks for CIFAR-10.

Original: 2 We develop a scaling law that predicts the error (when pruned) of an entire family of networks with varying widths and—in the case of the CIFAR-10 ResNets—depths.
Revised: 2 We develop a scaling law that predicts the error (when pruned) of an entire family of networks with varying widths and—in the case of the CIFAR-10 ResNets—depths.

Original: To vary width, we multiply the number of channels in each layer by a width scaling factor .
Revised: To vary width, we multiply the number of channels in each layer by a width scaling factor .

Original: To vary depth of the CIFAR-10 ResNets, we vary the number of residual blocks.
Revised: To vary depth of the CIFAR-10 ResNets, we vary the number of residual blocks.

Original: We refer to a network by its depth l (the number of layers in the network, not counting skip connections) and its width scaling factor w .
Revised: We refer to a network by its depth l (the number of layers in the network, not counting skip connections) and its width scaling factor w .

Original: Notation and terminology.
Revised: Notation and terminology.

Original: Throughout the paper we use the following notation and terminology:
Revised: Throughout the paper, we use the following notation and terminology:

Original: • D N = { x i , y i } Ni
Revised: • D N = { x i , y i } Ni

Original: =1 is a labeled training set with N examples.
Revised: =1 is a labeled training set with N examples.

Original: A subsample of size n is a subset of D N containing n examples selected uniformly at random.
Revised: A subsample of size n is a subset of D N containing n examples selected uniformly at random.

Original: • l and w are, respectively, the depth (i.e., the number of layers, excluding skip connections) and the width scaling factor of a particular network.
Revised: • l and w are, respectively, the depth (i.e., the number of layers, excluding skip connections) and the width scaling factor of a particular network.

Original: • A collection of networks that vary by width and depth are a network family .
Revised: • A collection of networks that vary by width and depth are a network family .

Original: • s is the sparsity of a pruned network (i.e., the fraction of weights that have been pruned) and d (cid:44) 1 − s is the density (i.e., the fraction of weights that have not been pruned).
Revised: • s is the sparsity of a pruned network (i.e., the fraction of weights that have been pruned) and d (cid:44) 1 − s is the density (i e., the fraction of weights that have not been pruned).

Original: • (cid:15) ( d, l, w, n ) is the test error of a network with the speciﬁed density, depth, width scaling factor, and dataset size.
Revised: • (cid:15) ( d, l, w, n ) is the test error of a network with the speciﬁed density, depth, width scaling factor, and dataset size.

Original: • (cid:15) np
Revised: • (cid:15) np

Original: ( l, w, n ) = (cid:15) (1 , l, w, n ) is the test error of the unpruned network with the speciﬁed depth, width scaling factor, and dataset size.
Revised: ( l, w, n ) = (cid:15) (1 , l, w, n ) is the test error of the unpruned network with the speciﬁed depth, width scaling factor, and dataset size.

Original: When clear from context, we omit ( w, l, n ) and write (cid:15) np.
Revised: When clear from context, we omit ( w, l, n ) and write (cid:15) np

Original: • ˆ (cid:15) ( (cid:15) np , d | l, w, n ) is an estimate of the error of a pruned model for a scaling law that has been ﬁt to a speciﬁc network with the speciﬁed depth, width scaling factor, and dataset size (Section 3).
Revised: • ˆ (cid:15) ( (cid:15) np , d | l, w, n ) is an estimate of the error of a pruned model for a scaling law that has been ﬁt to a speciﬁc network with the speciﬁed depth, width scaling factor, and dataset size (Section 3).

Original: • ˆ (cid:15) ( (cid:15) np , d, l, w, n ) is an estimate of the error of a pruned model with the speciﬁed depth, width scaling factor, and dataset size for a scaling law that has been ﬁt to a network family (Section 4).
Revised: • ˆ (cid:15) ( (cid:15) np , d, l, w, n ) is an estimate of the error of a pruned model with the speciﬁed depth, width scaling factor, and dataset size for a scaling law that has been ﬁt to a network family (Section 4).

Original: Dimensions.
Revised: Dimensions.

Original: In developing scaling laws, we vary four different dimensions: dataset subsample size ( n ) and network degrees of freedom density ( d ), network depth ( l ), and width scaling factor ( w ).
Revised: In developing scaling laws, we vary four different dimensions: dataset subsample size ( n ) and network degrees of freedom density ( d ), network depth ( l ), and width scaling factor ( w ).

Original: We consider the following ranges of these values in our experiments in the main body of the paper:
Revised: We consider the following ranges of these values in our experiments in the main body of the paper:

Original: [Table]
Revised: [Table]

Original: We use sanity checks to ﬁlter infeasible or unusual conﬁgurations from this table.
Revised: We use sanity checks to ﬁlter infeasible or unusual conﬁgurations from this table.

Original: In many cases, networks become disconnected before we reach the lowest density (e.g., ∼ 30% of the CIFARResNet conﬁgurations).
Revised: In many cases, networks become disconnected before we reach the lowest density (e.g., ∼ 30% of the CIFARResNet conﬁgurations).

Original: We also eliminate conﬁgurations where increasing width or depth of the unpruned network lowers test accuracy (e.g., 144 of the 294 CIFAR-10 ResNet conﬁgurations of l , w , and n ); these are typically unusual, imbalanced conﬁgurations (e.g., l = 98 , w = 116 ).
Revised: We also eliminate conﬁgurations where increasing width or depth of the unpruned network lowers test accuracy (e.g., 144 of the 294 CIFAR-10 ResNet conﬁgurations of l , w , and n ); these are typically unusual, imbalanced conﬁgurations (e.g., l = 98 , w = 1 16 ).

Original: Of the 12, possible CIFAR-10 ResNet conﬁgurations, about 8,000 are eliminated based on these sanity checks.
Revised: Of the 12, possible CIFAR-10 ResNet conﬁgurations, about 8,000 are eliminated based on these sanity checks.

Original:  M ODELING THE E RROR OF A P RUNED N ETWORK
Revised:  M ODELING THE E RROR OF A P RUNED N ETWORK

Original: Our goal in this section is to develop a functional form that accurately models the error of a member of a network family as it is pruned (using IMP here and SynFlow in Appendix E) based on its unpruned error (cid:15)
Revised: Our goal in this section is to develop a functional form that accurately models the error of a member of a network family as it is pruned (using IMP here and SynFlow in Appendix E) based on its unpruned error (cid:15)

Original: np ( w, l, n ) .
Revised: np ( w, l, n ) .

Original: In other words, we wish to ﬁnd a function ˆ (cid:15) ( (cid:15) np , d | l, w, n ) that predicts the error at each density d for a network with a particular depth l , width scaling factor w , and dataset size n .
Revised: In other words, we wish to ﬁnd a function ˆ (cid:15) ( (cid:15) np , d | l, w, n ) that predicts the error at each density d for a network with a particular depth l , width scaling factor w , and dataset size n .

Original: Intuition.
Revised: Intuition.

Original: Since IMP prunes a network 20% at a time, it produces pruned networks at intermediate levels of density d k = 0 .
Revised: Since IMP prunes a network 20% at a time, it produces pruned networks at intermediate levels of density d k = 0 .

Original: 8 k in the process of creating a ﬁnal pruned network at density d K = 0 .
Revised: 8 k in the process of creating a ﬁnal pruned network at density d K = 0 .

Original: 8 K .
Revised: 8 K .

Original: In Figure 1 (left), we plot the error of these pruned networks for members of the CIFAR-10 ResNet family with a different widths w .
Revised: In Figure 1 (left), we plot the error of these pruned networks for members of the CIFAR-10 ResNet family with a different widths w .

Original: All of these curves follow a similar pattern: 
Revised: All of these curves follow a similar pattern: [Figure] + [Figure]

Original: Observation 1: Low-error plateau.
Revised: Observation 1: Low-error plateau.

Original: The densest pruned networks (right part of the curves) have approximately the same error as the unpruned network: (cid:15)
Revised: The densest pruned networks (right part of the curves) have approximately the same error as the unpruned network: (cid:15)

Original: np ( w ) .
Revised: np ( w ) .

Original: We call this the low-error plateau .
Revised: We call this the low-error plateau .

Original: [Figure]

Original: [Figure]

Original: i m a t e d e rr o r fit

Original: [Figure]

Original: Observation 2: Power-law region.
Revised: Observation 2: Power-law region.

Original: When pruned further, error increases in a linear fashion on the logarithmic axes of the ﬁgure.
Revised: When pruned further, error increases in a linear fashion on the logarithmic axes of the ﬁgure.

Original: Linear behavior on a logarithmic scale is the functional form of a power law , in which error relates to density through an exponent γ and a coefﬁcient c : (cid:15) ( d, w ) ≈ cd − γ .
Revised: Linear behavior on a logarithmic scale is the functional form of a power law , in which error relates to density through an exponent γ and a coefﬁcient c : (cid:15) ( d, w ) ≈ cd − γ .

Original: In particular, γ controls the slope of the line on the logarithmic axes.
Revised: In particular, γ controls the slope of the line on the logarithmic axes.

Original: Observation 3: High-error plateau.
Revised: Observation 3: High-error plateau.

Original: When pruned further, error again ﬂattens; we call this the high-error plateau and call the error of the plateau (cid:15) ↑ .
Revised: When pruned further, error again ﬂattens; we call this the high-error plateau and call the error of the plateau (cid:15) ↑ .

Original: Figure 1 (center) labels these regions for CIFAR-10 ResNet-20 (width scaling factor1, dataset size
Revised: Figure 1 (center) labels these regions for CIFAR-10 ResNet 20 (width scaling factor dataset size

Original: N )
Revised: N )

Original: and shows an approximation of these regions that is piece-wise linear on logarithmic axes.
Revised: and shows an approximation of these regions that is piece-wise linear on logarithmic axes.

Original: These observations are our starting point for developing a functional form that estimates error when pruning.
Revised: These observations are our starting point for developing a functional form that estimates error when pruning.

Original: Functional form.
Revised: Functional form.

Original: Our next task is to ﬁnd a functional form that accurately captures these observations about the relationship between density and error.
Revised: Our next task is to ﬁnd a functional form that accurately captures these observations about the relationship between density and error.

Original: In prior work, Rosenfeld et al.
Revised: In prior work, Rosenfeld et al.

Original: (2020) observe that the relationship between width and error shares the same general shape: it has a region of lower error, a power-law region, and region of higher error.
Revised: (2020) observe that the relationship between width and error shares the same general shape: it has a region of lower error, a power-law region, and region of higher error.

Original: However, this relationship is different enough from the one we observe (see Appendix H) to merit an entirely new functional form.
Revised: However, this relationship is different enough from the one we observe (see Appendix G) to merit an entirely new functional form.

Original: To develop this functional form, we note that the three regions of the curves in Figure 1 (the low-error plateau, the power-law region, and the high-error plateau) can be described by three power laws: two plateaus with exponent zero and one intermediate region with exponent γ .
Revised: To develop this functional form, we note that the three regions of the curves in Figure 1 (the low-error plateau, the power-law region, and the high-error plateau) can be described by three power laws: two plateaus with exponent zero and one intermediate region with exponent γ .

Original: A functional family that arises frequently in the context of systems that exhibit different power-law regions is the rational family .
Revised: A functional family that arises frequently in the context of systems that exhibit different power-law regions is the rational family .

Original: The particular family member we consider is as follows:
Revised: The particular family member we consider is as follows:

Original: [Equation]
Revised: [Equation]

Original: where j
Revised: This function’s shape is controlled by (cid:15) np , (cid:15) ↑ , γ , and p (visualized in Figure 1, right).

Original: [Equation]
Revised: (cid:15) np and (cid:15) ↑ are the values of the low and high-error plateaus.

Original: √− [Equation] [Equation]
Revised: γ is the slope of the power-law region on logarithmic axes.

Original: [Figure]
Revised: p controls the density at which the high-error plateau transitions to the power-law region.

Original: Fitting.
Revised: Fitting.

Original: To ﬁt ˆ (cid:15) ( (cid:15) np , d | l, w, n ) to the actual data (cid:15) ( d, l, w, n ) , we estimate values for the free parameters (cid:15) ↑ , γ , and p by minimizing the relative error δ (cid:44) ˆ (cid:15) ( (cid:15) np ,d | l,w,n ) − (cid:15) ( d,l,w,n ) (cid:15) ( d,l,w,n  using least squares regression.
Revised: To ﬁt ˆ (cid:15) ( (cid:15) np , d | l, w, n ) to the actual data (cid:15) ( d, l, w, n ) , we estimate values for the free parameters (cid:15) ↑ , γ , and p by minimizing the relative error δ (cid:44) ˆ (cid:15) ( (cid:15) np ,d | l,w,n ) − (cid:15) ( d,l,w,n ) (cid:15) ( d,l,w,n ) using least squares regression.

Original: The ﬁt is performed separately for each conﬁguration ( l, w, n ) for all 30– densities, resulting in per-conﬁguration 
Revised: The ﬁt is performed separately for each conﬁguration ( l, w, n ) for all 30– densities, resulting in per-conﬁguration estimates of ˆ (cid:15) ↑ , ˆ γ , and ˆ p .

Original: Evaluating ﬁt.
Revised: Evaluating ﬁt.

Original: For a qualitative view, 5 we plot the actual error 6 (cid:15) ( d, l, w, n ) and the estimated error ˆ (cid:15) ( (cid:15) np , d | l, w, n ) as a function density for CIFAR-10 ResNets of varying widths (Figure 2, left). 
Revised: For a qualitative view, 5 we plot the actual error 6 (cid:15) ( d, l, w, n ) and the estimated error ˆ (cid:15) ( (cid:15) np , d | l, w, n ) as a function density for CIFAR-10 ResNets of varying widths (Figure 2, left). Our [Figure] fit :1.1%

Original: Our estimated error appears to closely follow the actual error.
Revised: [Figure] [Figure] estimated error appears to closely follow the actual error.

Original: The most noticeable deviations occur at large densities, where the error dips slightly when pruning whereas we treat it as ﬂat (see Section 5).
Revised: The most noticeable deviations occur at large densities, where the error dips slightly when pruning whereas we treat it as ﬂat (see Section 6).

Original: Quantitatively, we measure the extent to which estimated error departs from the actual error using the mean µ and standard deviation σ of the relative deviation δ .
Revised: Quantitatively, we measure the extent to which estimated error departs from the actual error using the mean µ and standard deviation σ of the relative deviation δ .

Original: Figure 2 (center) compares the estimated and actual errors for the networks in Figure 2 (left); Figure 2 (right) shows the same comparison for all conﬁgurations of l , w , and n on CIFAR-10 and the more than 4000 pruned ResNets that result.
Revised: Figure 2 (center) compares the estimated and actual errors for the networks in Figure 2 (left); Figure 2 (right) shows the same comparison for all conﬁgurations of l , w , and n on CIFAR-10 and the more than 4000 pruned ResNets that result.

Original: The relative deviation on all conﬁgurations has mean µ < 2% and standard deviation σ < 4% ; this means that, if the actual error is 10% , the estimated error is 9 . 8 ± 0 .
Revised: The relative deviation on all conﬁgurations has mean µ < 2% and standard deviation σ < 4% ; this means that, if the actual error is 10% , the estimated error is 9 . 8 ± 0 .

Original: 4% ( ˆ (cid:15) =
Revised: 4% ( ˆ (cid:15) =

Original: (1 − δ ) (cid:15) ).
Revised: (1 − δ ) (cid:15) ).

Original:  J OINTLY M ODELING E RROR A CROSS A LL D IMENSIONS
Revised:  J OINTLY M ODELING E RROR A CROSS A LL D IMENSIONS

Original: In Section 3, we found a functional form ˆ (cid:15) ( (cid:15) np , d | l, w, n ) (Equation 1) that accurately predicts the error when pruning a speciﬁc member of a network family (with depth l and width w ) trained with a dataset of size n .
Revised: In Section 3, we found a functional form ˆ (cid:15) ( (cid:15) np , d | l, w, n ) (Equation 1) that accurately predicts the error when pruning a speciﬁc member of a network family (with depth l and width w ) trained with a dataset of size n .

Original: The parameters governing Equation 1 ( (cid:15) ↑ , p , and γ ) were allowed to vary between different conﬁgurations and depend on l , w , n .
Revised: The parameters governing Equation 1 ( (cid:15) ↑ , p , and γ ) were allowed to vary between different conﬁgurations and depend on l , w , n .

Original: However, we are interested in a single joint scaling law ˆ (cid:15) ( (cid:15) np , d, l, w, n ) that, given the unpruned network error (cid:15) np ( l, w, n ) , accurately predicts error across all dimensions we consider: all members of a network family that vary in depth and width, all densities, and all dataset sizes.
Revised: However, we are interested in a single joint scaling law ˆ (cid:15) ( (cid:15) np , d, l, w, n ) that, given the unpruned network error (cid:15) np ( l, w, n ) , accurately predicts error across all dimensions we consider: all members of a network family that vary in depth and width, all densities, and all dataset sizes.

Original: Importantly, the parameters of such a joint scaling law must be constants as a function of all dimensions.
Revised: Importantly, the parameters of such a joint scaling law must be constants as a function of all dimensions.

Original: In this section, we develop this joint scaling law.
Revised: In this section, we develop this joint scaling law.

Original: Intuition: the error-preserving invariant.
Revised: Intuition: the error-preserving invariant.

Original: Our desired scaling law ˆ (cid:15) ( (cid:15) np , d, l, w, n ) will be a fourdimensional function of d , w , l , and n .
Revised: Our desired scaling law ˆ (cid:15) ( (cid:15) np , d, l, w, n ) will be a fourdimensional function of d , w , l , and n .

Original: To develop this, we study the interdependence between density and depth or width by examining two-dimensional projections of the actual error (cid:15) ( d, l, w, n ) (Figure 3).
Revised: To develop this, we study the interdependence between density and depth or width by examining two-dimensional projections of the actual error (cid:15) ( d, l, w, n ) (Figure 3).

Original: These plots display contours of constant error as density and depth or width vary.
Revised: These plots display contours of constant error as density and depth or width vary.

Original: Consider the projection onto the plane of density and depth (Figure 3, left).
Revised: Consider the projection onto the plane of density and depth (Figure 3, left).

Original: The constant-error contours are linear except for in the densest networks, meaning each contour traces a power-law relationship between d and l .
Revised: The constant-error contours are linear except for in the densest networks, meaning each contour traces a power-law relationship between d and l .

Original: In other words, we can describe all combinations of densities and widths that produce error (cid:15) v using l φ d = v , where v is a constant at which network error is (cid:15) v and φis the slope of the contour on the logarithmic axes. The contours of density and width also have this
Revised: In other words, we can describe all combinations of densities and widths that produce error (cid:15) v using l φ d = v , where v is a constant at which network error is (cid:15) v and φ

Original: 3 log10( m* ) 1.
Revised: 3 log10( m* ) 1.

Original: l o g( t o pe rr ) error vs m*
Revised: l o g( t o pe rr ) error vs m*

Original: slope: p      3 log10( m* ) 1.
Revised: slope: p      3 log10( m* ) 1.

Original: l o g( t o pe rr ) error vs m*
Revised: l o g( t o pe rr ) error vs m*

Original: slope: p      3 log10( m* ) 1.
Revised: slope: p      3 log10( m* ) 1.

Original: l o g( t o pe rr ) error vs m*
Revised: l o g( t o pe rr ) error vs m*

Original: slope: p  pattern (Figure 3, right), meaning we can describe a similar relationship w ψ d = v (cid:48) .
Revised: slope: p is the slope of the contour on the logarithmic axes. The contours of density and width also have this pattern (Figure 3, right), meaning we can describe a similar relationship w ψ d = v (cid:48) .

Original: Finally, we can combine these observations about depth and width into the expression l φ w ψ d = v (cid:48)(cid:48) .
Revised: Finally, we can combine these observations about depth and width into the expression l φ w ψ d = v (cid:48)(cid:48) .

Original: We refer to the expression l φ w ψ d as the error-preserving invariant , and we denote it m ∗ .
Revised: We refer to the expression l φ w ψ d as the error-preserving invariant , and we denote it m ∗ .

Original: This invariant captures the observation that there exist many interchangeable combinations of depth, width, and density that achieve the same error and tells us which combinations do so.
Revised: This invariant captures the observation that there exist many interchangeable combinations of depth, width, and density that achieve the same error and tells us which combinations do so.

Original: For example, networks of vastly different densities reach the same error if we vary l and w according to the invariant.
Revised: For example, networks of vastly different densities reach the same error if we vary l and w according to the invariant.

Original: Functional form.
Revised: Functional form.

Original: The invariant allows us to convert the functional form ˆ (cid:15) ( (cid:15) np , d | l, w, n ) for a speciﬁc l , w , and n from Section 3 into a joint functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) for all l , w , and n .
Revised: The invariant allows us to convert the functional form ˆ (cid:15) ( (cid:15) np , d | l, w, n ) for a speciﬁc l , w , and n from Section 3 into a joint functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) for all l , w , and n .

Original: Rewriting the deﬁnition of the invariant, d = m ∗ l φ w ψ .
Revised: Rewriting the deﬁnition of the invariant, d = m ∗ l φ w ψ .

Original: We can substitute this for d in the functional form from Section 3.
Revised: We can substitute this for d in the functional form from Section 3.

Original: Finally, by rewriting p as p (cid:48)
Revised: Finally, by rewriting p as p (cid:48)

Original: l φ w ψ and canceling, we arrive at the expression:ˆ
Revised: l φ w ψ and canceling, we arrive at the expression:

Original: [Equation]
Revised: [Equation]

Original: ( (cid:15) np [Equation] d
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: l

Original: [Equation]
Revised: [Table]

Original: w

Original: [Equation]
Revised: [Equation]

Original: n )

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] which is the joint functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) of all four dimensions d , l , w , and n .
Revised: which is the joint functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) of all four dimensions d , l , w , and n .

Original: Critically, for this to be a useful joint form, the free parameters e ↑ , p (cid:48) , and γ must be constants shared across all possible values of d , l , w , and n .
Revised: Critically, for this to be a useful joint form, the free parameters e ↑ , p (cid:48) , and γ must be constants shared across all possible values of d , l , w , and n .

Original: We will assume this is the case and directly quantify how well this assumption holds in the ﬁt section to follow.
Revised: We will assume this is the case and directly quantify how well this assumption holds in the ﬁt section to follow.

Original: To glean some qualitative intuition as to why this may be a reasonable assumption, we can examine the relationship between m ∗ and the generalization error of pruned networks as we vary depth, width, and dataset size (Figure 4).
Revised: To glean some qualitative intuition as to why this may be a reasonable assumption, we can examine the relationship between m ∗ and the generalization error of pruned networks as we vary depth, width, and dataset size (Figure 4).

Original: Across all projections, the annotated e ↑ (error of the high-error plateau), γ (slope of the power-law region) and p (cid:48) (value of m ∗ where the high-error plateau transitions to the power-law region) are the same.
Revised: Across all projections, the annotated e ↑ (error of the high-error plateau), γ (slope of the power-law region) and p (cid:48) (value of m ∗ where the high-error plateau transitions to the power-law region) are the same.

Revised: Note that in Eq.the dependence on n is implicit, through (cid:15) np .

Revised: We retain the explicit form ˆ (cid:15) ( ..., n ) to stress that the lack of explicit dependency on n is non-trivial and was not known prior to our work.

Original: Fitting.
Revised: Fitting.

Original: To ﬁt ˆ (cid:15) ( (cid:15) np , d, l, w, n ) to the actual data (cid:15) ( d, l, w, n ) , we estimate values for the free parameters (cid:15) ↑ , γ, p (cid:48) , φ and ψ by minimizing the relative error δ (cid:44) ˆ (cid:15) ( (cid:15) np ,d,l,w,n ) − (cid:15) ( d,l,w,n ) (cid:15) ( d,l,w,n ) using least squares regression.
Revised: To ﬁt ˆ (cid:15) ( (cid:15) np , d, l, w, n ) to the actual data (cid:15) ( d, l, w, n ) , we estimate values for the free parameters (cid:15) ↑ , γ, p (cid:48) , φ and ψ by minimizing the relative error δ (cid:44) ˆ (cid:15) ( (cid:15) np ,d,l,w,n ) − (cid:15) ( d,l,w,n ) (cid:15) ( d,l,w,n ) using least squares regression.

Original: The ﬁt is performed jointly over all conﬁgurations of d, l, w, and n , resulting in joint estimates of ˆ (cid:15) ↑ , ˆ γ, ˆ p, ˆ φ, and ˆ ψ .
Revised: The ﬁt is performed jointly over all conﬁgurations of d, l, w, and n , resulting in joint estimates of ˆ (cid:15) ↑ , ˆ γ, ˆ p, ˆ φ, and ˆ ψ .

Original: One can also perform a partial ﬁt for a subset of dimensions (e.g., just d , l , and n ) by omitting φ and/or ψ (see Appendix D).
Revised: One can also perform a partial ﬁt for a subset of dimensions (e.g., just d , l , and n ) by omitting φ and/or ψ (see Appendix D).

Original: Evaluating ﬁt.
Revised: Evaluating ﬁt.

Original: In Figure 5, we plot the actual error (cid:15) ( d, l, w, n ) and the estimated error ˆ (cid:15) ( (cid:15) np , d, l, w, n ) for the CIFAR-10 ResNets and ImageNet ResNets (single depth).
Revised: In Figure 5, we plot the actual error (cid:15) ( d, l, w, n ) and the estimated error ˆ (cid:15) ( (cid:15) np , d, l, w, n ) for the CIFAR-10 ResNets and ImageNet ResNets (single depth).

Original: As in Section 3, our estimated error appears to closely follow the actual error.
Revised: As in Section 3, our estimated error appears to closely follow the actual error.

Original: Deviations arise mainly at high densities where error dips below (cid:15) np and low densities approaching high error saturation.
Revised: Deviations arise mainly at high densities where error dips below (cid:15) np and low densities approaching high error saturation.

Original: We again quantify the ﬁt of the estimated error using the mean µ and standard deviation σ of the relative deviation δ .
Revised: We again quantify the ﬁt of the estimated error using the mean µ and standard deviation σ of the relative deviation δ .

Original: The relative deviation on the joint scaling laws for the CIFAR-10 and Imagenet networks has a mean µ < 2% and standard deviation of σ < 6% .
Revised: The relative deviation on the joint scaling laws for the CIFAR-10 and Imagenet networks has a mean µ < 2% and standard deviation of σ < 6% .

Original: To contextualize these results, Figure 5 (right) quantiﬁes the variation in error we see over multiple trials of the CIFAR-10 experiments due to using different random seeds. 
Revised: To contextualize these results, Figure 5 (right) quantiﬁes the variation in error we see over multiple trials of the CIFAR-10 experiments due to using different random seeds. It plots the minimum, 1.0 actual error (mean) 0. e s t

Revised: i m a t e d e rr o r fit :1.2% :2.6% 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 actual error (mean) 0.

Original: It plots the minimum,{+e s t i m a t e d e rr o r fit µ :0.9% σ :5.8% 0.2 0.
 0.2 0.4 0.6 0.8 1.0 1.2 actual error (mean) 0. a c t u a l e rr o r ( n o m e a n ) baseline µ :0.0% σ :3.4%+} maximum, and mean errors across the three trials we ran.
Revised: {+e s t i m a t e d e rr o r fit µ :0.9% σ :5.8% 0.2 0.
 0.2 0.4 0.6 0.8 1.0 1.2 actual error (mean) 0. a c t u a l e rr o r ( n o m e a n ) baseline µ :0.0% σ :3.4%+} maximum, and mean errors across the three trials we ran.

Original: 7 The variation across trials has a standard deviation of σ = 3 .
Revised: 7 The variation across trials has a standard deviation of σ = 3 .

Original: 4% , sizeable relative to the estimation error of σ = 5 . 8% for the joint scaling law.
Revised: 4% , sizeable relative to the estimation error of σ = 5 . 8% for the joint scaling law.

Original: This indicates that a signiﬁcant portion of our error may stem from measurement noise.[Figure] e s t i m a t e d e rr o r fit [Figure]
Revised: This indicates that a signiﬁcant portion of our error may stem from measurement noise.

Original: [Figure]

Original: a c t u a l e rr o r ( n o m e a n ) baseline µ :0.0% σ :3.4%

Original: [Figure]

Original: The functional form has just ﬁve parameters and obtains an accurate ﬁt on over 4000 points, suggesting it is a good approximation.
Revised: The functional form has just ﬁve parameters and obtains an accurate ﬁt on over 4000 points, suggesting it is a good approximation.

Original: In Appendix E, we show that it achieves a similarly good ﬁt for VGG-style networks and for the SynFlow pruning algorithm.
Revised: In Appendix E, we show that it achieves a similarly good ﬁt for VGG-style networks and for the SynFlow pruning algorithm.

Original: In Appendix F, we show that it is possible to get a good ﬁt with far fewer points and that the ﬁt has low sensitivity to the choice of points.
Revised: In Section 5, we show that it is possible to get a good ﬁt with far fewer points and that the ﬁt has low sensitivity to the choice of points.

Original: P RINCIPLES FOR S ELECTING  A  F UNCTIONAL F AMILY
Revised:  A NALYZING THE S ENSITIVITY OF THE F IT TO N UMBER OF P OINTS

Revised: In Section 4, we showed that our scaling law was accurate when we ﬁt it on all of the available data.

Revised: Now that we possess the functional form and know that it can accurately model the behavior of IMP, we study the amount of data necessary to obtain a stable, 8 accurate ﬁt.

Revised: This question is especially relevant when the functional form is applied to new settings—new networks, datasets, or pruning algorithms—and we must collect new data to do so.

Revised: The functional form has only ﬁve parameters, suggesting that few experiments will be necessary.

Revised: Experiments.

Revised: To evaluate the effect of the number of points on the stability and accuracy of the ﬁt, we randomly sample varying numbers of points, ﬁt the scaling law to those points, and evaluate the quality of the ﬁt over all points.

Revised: We sample these points in two ways.

Revised: Experiment 1.

Revised: Randomly sample T network conﬁgurations ( w, l, n, d ) .

Revised: This experiment captures the use case of algorithms such as SynFlow (Appendix E), where obtaining data at any density d relies only on possessing the unpruned network, not other densities d (cid:48) > d .

Revised: Experiment 2.

Revised: Randomly sample T network conﬁgurations ( w, l, n ) and include all densities d for each conﬁguration.

Revised: This experiment captures the use case of algorithms such as IMP, where obtaining data at density d requires obtaining all densities d (cid:48) > d .

Revised: As such, we anticipate that data will be obtained by iteratively pruning a small number of conﬁgurations ( w, l, n ) to low density.

Revised: Results.

Revised: We perform each experiment for many different values of T on the CIFAR-10 ResNets pruned with IMP.

Revised: We repeat the experiment at each value of T 30 times (with a different sample of points) and report the mean and standard deviation of µ and σ for the ﬁt.

Revised: Experiments 1 and 2 appear in Figure 6.

Revised: The shaded areas represent one standard deviation from the mean in each direction.

Revised: On

Revised: Experiment 1, when just 40 conﬁgurations of ( w, l, d, n ) are available, the standard deviation on both µ and σ is just one percentage point.

Revised: On Experiment 2, when just 15 random conﬁgurations of ( w, l, n ) are available at all densities, we similarly achieve standard deviation below 1%.

Revised: In both cases, as the number of networks increases, the standard deviation decreases further.

Revised: [Figure]

Revised: [Figure]

Revised: These results show that, now that our scaling law is known, it is possible to obtain an accurate (and stable) estimation using far less data than we used to evaluate the quality of the ﬁt in the main body of the paper.

Revised: Importantly, the experiments we perform in this section are particularly naive.

Revised: We make no effort to ensure that the conﬁgurations we select represent a diverse range of widths, depths, dataset sizes

Revised: [Figure]

Revised: and densities.

Revised: By selecting these conﬁgurations in a strategic way, we believe it would be possible to further reduce the number of conﬁgurations necessary to obtain a similarly accurate ﬁt.

Revised:  P RINCIPLES FOR S ELECTING A F UNCTIONAL F AMILY

Original: In this section, we discuss some of the key criteria that led us to select this particular functional form and opportunities for further reﬁnement.
Revised: In this section, we discuss some of the key criteria that led us to select this particular functional form and opportunities for further reﬁnement.

Original: Criterion 1: Transitions.
Revised: Criterion 1: Transitions.

Original: In Section 3, we observe that, when pruning a neural network, error has a low-error plateau, a power-law region, and a high-error plateau.
Revised: In Section 3, we observe that, when pruning a neural network, error has a low-error plateau, a power-law region, and a high-error plateau.

Original: Between these regions are transitions where error varies smoothly from one region to the next.
Revised: Between these regions are transitions where error varies smoothly from one region to the next.

Original: Matching the shape of these transitions was a key consideration for selecting our function family.
Revised: Matching the shape of these transitions was a key consideration for selecting our function family.

Original: To illustrate the importance of properly ﬁtting the transitions, Figure 6 (left) shows two possible functional families for ﬁtting the relationship between density and error for CIFAR-10 ResNets.
Revised: To illustrate the importance of properly ﬁtting the transitions, Figure 7 (left) shows two possible functional families for ﬁtting the relationship between density and error for CIFAR-10 ResNets.

Original: Actual error is in gray, and the functional form from Section 3 is in blue.
Revised: Actual error is in gray, and the functional form from Section 3 is in blue.

Original: In red is the ﬁt for a functional form adapted from the one that Rosenfeld et al.
Revised: In red is the ﬁt for a functional form adapted from the one that Rosenfeld et al.

Original: (2020) use to model the relationship between width and error.
Revised: (2020) use to model the relationship between width and error.

Original: The difference between these functional families is the way they model transitions, and the one we choose in this paper better models the transitions in our setting.
Revised: The difference between these functional families is the way they model transitions, and the one we choose in this paper better models the transitions in our setting.

Original: For further discussion of this comparison, see Appendix H.
Revised: For further discussion of this comparison, see Appendix G.

Original: Criterion 2: A small number of interpretable parameters.
Revised: Criterion 2: A small number of interpretable parameters.

Original: Selecting a functional form is not merely a curve-ﬁtting exercise.
Revised: Selecting a functional form is not merely a curve-ﬁtting exercise.

Original: We seek the underlying structure that governs the relationships between d, l, w, n , and error in a manner akin to a law of physics.
Revised: We seek the underlying structure that governs the relationships between d, l, w, n , and error in a manner akin to a law of physics.

Original: As such, our functional form should have a small number of parameters that are interpretable .
Revised: As such, our functional form should have a small number of parameters that are interpretable .

Original: In our functional form (Equation 2), each parameter has a clear meaning.
Revised: In our functional form (Equation 2), each parameter has a clear meaning.

Original: The parameters (cid:15) ↑ , p (cid:48) , and γ control the high-error plateau, the transition to the power-law region, and the slope of the power-law region.
Revised: The parameters (cid:15) ↑ , p (cid:48) , and γ control the high-error plateau, the transition to the power-law region, and the slope of the power-law region.

Original: φ and ψ control the interchangeability of width and depth with density.
Revised: φ and ψ control the interchangeability of width and depth with density.

Original: We approximate error over multiple orders of magnitude and over 4,000 conﬁgurations of ResNet-20 on CIFAR-10 with just ﬁve parameters, indicating we have distilled key information about the behavior of pruning into our functional form.
Revised: We approximate error over multiple orders of magnitude and over 4,000 conﬁgurations of ResNet-20 on CIFAR-10 with just ﬁve parameters, indicating we have distilled key information about the behavior of pruning into our functional form.

Original: Sources of systemic error and limitations of our approximation form.
Revised: Sources of systemic error and limitations of our approximation form.

Original: By seeking to minimize the number of parameters in our functional form, we leave some phenomena unmodeled.
Revised: By seeking to minimize the number of parameters in our functional form, we leave some phenomena unmodeled.

Original: In particular, there are two phenomena we have chosen not to model that introduce systemic error.
Revised: In particular, there are two phenomena we have chosen not to model that introduce systemic error.

Original: First, the low-error plateau is not a plateau.
Revised: First, the low-error plateau is not a plateau.

Original: Error often improves slightly at high densities before returning to (cid:15)
Revised: Error often improves slightly at high densities before returning to (cid:15)

Original: np during the transition to the power-law region.
Revised: np during the transition to the power-law region.

Original: Our model treats the region as ﬂat and treats error as monotonically increasing as density decreases.
Revised: Our model treats the region as ﬂat and treats error as monotonically increasing as density decreases.

Original: This source of error accounts for a bias of ∼ 1% relative error in our estimation (Appendix I).
Revised: This source of error accounts for a bias of ∼ 1% relative error in our estimation (Appendix H).

Original: Second, we model both transitions (between the power-law region and each plateau) with a single shape and the same transition rate.
Revised: Second, we model both transitions (between the power-law region and each plateau) with a single shape and the same transition rate.

Original: If we treated each transition separately and used higher-order terms in the rational form, we could potentially reduce some of the residual error in our estimation at the cost of additional complexity.
Revised: If we treated each transition separately and used higher-order terms in the rational form, we could potentially reduce some of the residual error in our estimation at the cost of additional complexity.

Original:  I MPLICATIONS AND C ONCLUSIONS
Revised:  I MPLICATIONS AND C ONCLUSIONS

Original: Our main contribution is a functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) that accurately predicts the error when pruning members of a network family using both IMP and SynFlow.
Revised: Our main contribution is a functional form ˆ (cid:15) ( (cid:15) np , d, l, w, n ) that accurately predicts the error when pruning members of a network family using both IMP and SynFlow.

Original: There are several broader implications of our ability to characterize pruning in this way.
Revised: There are several broader implications of our ability to characterize pruning in this way.

Original: The mere existence of this functional form means there is indeed structure to the way pruning affects error.
Revised: The mere existence of this functional form means there is indeed structure to the way pruning affects error.

Original: Although prior work (Cai et al., 2019) has implicitly relied on such structure, we are the ﬁrst to explicitly describe it.
Revised: Although prior work (Cai et al., 2019) has implicitly relied on such structure, we are the ﬁrst to explicitly describe it.

Original: This functional form enables a framework in which we can reason conceptually and analytically about pruning.
Revised: This functional form enables a framework in which we can reason conceptually and analytically about pruning.

Original: In doing so, we can make new observations about pruning that are non-obvious or costly to exhaustively demonstrate empirically.
Revised: In doing so, we can make new observations about pruning that are non-obvious or costly to exhaustively demonstrate empirically.

Original: For example, recall our motivating question:
Revised: For example, recall our motivating question:

Original: Given a family of neural networks, which should we prune (and by how much) to obtain the network with the smallest parameter-count such that its error does not exceed some threshold (cid:15) k ?
Revised: Given a family of neural networks, which should we prune (and by how much) to obtain the network with the smallest parameter-count such that its error does not exceed some threshold (cid:15) k ?

Original: This is an optimization problem—ﬁnd the conﬁguration of d , l , and w that minimizes parameter-count subject to an error constraint: argmin w,l,d m s.t.
Revised: This is an optimization problem—ﬁnd the conﬁguration of d , l , and w that minimizes parameter-count subject to an error constraint: argmin w,l,d m s.t.

Original: ˆ (cid:15) =
Revised: ˆ (cid:15) =

Original: (cid:15) k .
Revised: (cid:15) k .

Original: For ResNets m ∝ dlw 2 , yielding: l, w, d = argmin l,w,d lw 2 d s.t.
Revised: For ResNets m ∝ dlw 2 , yielding: l, w, d = argmin l,w,d lw 2 d s.t.

Original: [Equation]
Revised: [Equation]

Original: which is solvable directly without running any further experiments.
Revised: which is solvable directly without running any further experiments.

Original: Using this approach, we can derive a useful insight.
Revised: Using this approach, we can derive a useful insight.

Original: In the pruning literature, it is typical to report the minimum density at which the pruned network can match the error (cid:15)
Revised: In the pruning literature, it is typical to report the minimum density at which the pruned network can match the error (cid:15)

Original: np
Revised: np

Original: ( l, w ) of the unpruned network (Han et al., 2015).
Revised: ( l, w ) of the unpruned network (Han et al., 2015).

Original: However, our scaling law suggests that this is not the smallest model that achieves error (cid:15)
Revised: However, our scaling law suggests that this is not the smallest model that achieves error (cid:15)

Original: np ( l, w ) .
Revised: np ( l, w ) .

Original: Instead, it is better to train a larger network with depth l (cid:48) and width w (cid:48) and prune until error reaches (cid:15)
Revised: Instead, it is better to train a larger network with depth l (cid:48) and width w (cid:48) and prune until error reaches (cid:15)

Original: np ( l, w ) , even as that results in error well above (cid:15)
Revised: np ( l, w ) , even as that results in error well above (cid:15)

Original: np ( l (cid:48) , w (cid:48) ) .
Revised: np ( l (cid:48) , w (cid:48) ) .

Original: This analytic result parallels and extends the ﬁndings of Li et al.
Revised: This analytic result parallels and extends the ﬁndings of Li et al.

Original: (2020) on NLP tasks.
Revised: (2020) on NLP tasks.

Original: Figure 6 (center) illustrates this behavior: it shows the error predicted by our scaling law for CIFAR10 ResNets with varying widths.
Revised: Figure 7 (center) illustrates this behavior: it shows the error predicted by our scaling law for CIFAR10 ResNets with varying widths.

Original: The dotted black line shows the minimal parameter-count at which we predict it is possible to achieve each error.
Revised: The dotted black line shows the minimal parameter-count at which we predict it is possible to achieve each error.

Original: Importantly, none of the low-error plateaus intersect this black dotted line, meaning a model cannot be minimal until it has been pruned to the point where it increases in error.
Revised: Importantly, none of the low-error plateaus intersect this black dotted line, meaning a model cannot be minimal until it has been pruned to the point where it increases in error.

Original: This occurs because the transitions of our functional form are gradual.
Revised: This occurs because the transitions of our functional form are gradual.

Original: On the other hand, if we start with a model that is too large, it will no longer be on the black line when it has been pruned to the point where its error reaches (cid:15)
Revised: On the other hand, if we start with a model that is too large, it will no longer be on the black line when it has been pruned to the point where its error reaches (cid:15)

Original: np ( l, w ) .
Revised: np ( l, w ) .

Original: In Figure 6 (right), we plot the same information from the actual CIFAR-10 data and see the same phenomena occur in practice.
Revised: In Figure 7 (right), we plot the same information from the actual CIFAR-10 data and see the same phenomena occur in practice.

Original: The difference between the estimated and actual optimal parameter count is no more than 25%.
Revised: The difference between the estimated and actual optimal parameter count is no more than 25%.

Original: Looking ahead, there are several opportunities for future work.
Revised: Looking ahead, there are several opportunities for future work.

Original: Better understanding the sources of systematic error (error dips and transition shape) is a promising avenue for making it possible to extrapolate from small-scale settings to large-scale settings (see Appendix G for a forward looking discussion of extrapolation).
Revised: Better understanding the sources of systematic error (error dips and transition shape) is a promising avenue for making it possible to extrapolate from small-scale settings to large-scale settings (see Appendix F for a forward looking discussion of extrapolation).

Original: Furthermore, although we focus on pruning for image classiﬁcation and networks and pruning methods differ in other contexts (e.g., NLP), the generality of our functional form to different pruning strategies and network families suggests it may have broader applicability.
Revised: Furthermore, although we focus on pruning for image classiﬁcation and networks and pruning methods differ in other contexts (e.g., NLP), the generality of our functional form to different pruning strategies and network families suggests it may have broader applicability.

Original: C ONTENTS OF A PPENDICES
Revised: C ONTENTS OF A PPENDICES

Original: Appendix A. Details on the IMP and SynFlow pruning algorithms.
Revised: Appendix A. Details on the IMP and SynFlow pruning algorithms

Original: Appendix B. Details on the models, datasets, and traininghyperparameters.
Revised: Appendix B. Details on the models , datasets and training

Original: Appendix
Revised: hyperparameters.

Original: C Full data for the observations we make in Section 3.
Revised: Appendix C. Full data for the observations we make in Section 3.

Original: Appendix D. Partial ﬁts (e.g., just width, depth, or dataset size) for the joint scaling law in Section 4.
Revised: Appendix D. Partial ﬁts (e.g., just width, depth, or dataset size) for the joint scaling law in Section 4.

Original: Appendix E. Demonstrating that our functional form applies to VGG-style networks for CIFARand to the SynFlow pruning technique.
Revised: Appendix E. Demonstrating that our functional form applies to VGG-style networks for CIFARand to the SynFlow pruning technique.

Original: Appendix F. Demonstrating that, now that the functional form has been established, it is possible to obtain an accurate ﬁt with a small amount of data.
Revised: Appendix 5. Demonstrating that, now that the functional form has been established, it is possible to obtain an accurate ﬁt with a small amount of data.

Original: Appendix G. A discussion of extrapolation for our scaling law.
Revised: Appendix F. A discussion of extrapolation for our scaling law.

Original: Appendix H. A more detailed comparison between our scaling law and that of Rosenfeld et al.
Revised: Appendix G. A more detailed comparison between our scaling law and that of Rosenfeld et al.

Original: (2020) following up on Section 5.
Revised: (2020) following up on Section 6.

Original: Appendix I. How we computed the effect of error dips on our estimator in Section 5.
Revised: Appendix H. How we computed the effect of error dips on our estimator in Section 6.

Original: A P RUNING A LGORITHMS A. F ORMAL S TATEMENT OF I TERATIVE M AGNITUDE P RUNING (IMP)
Revised: A P RUNING A LGORITHMS A. F ORMAL S TATEMENT OF I TERATIVE M AGNITUDE P RUNING (IMP)

Original: A. S YN F LOW
Revised: A. S YN F LOW

Original: Unlike IMP, SynFlow is a pruning algorithm that prunes neural networks before any training has taken place (Tanaka et al., 2020).
Revised: Unlike IMP, SynFlow is a pruning algorithm that prunes neural networks before any training has taken place (Tanaka et al., 2020).

Original: To do so, SynFlow computes the “synaptic strengths” of each connection and prunes those weights with the lowest synaptic strengths (see Algorithm 2 below for the details on computing the synaptic strengths).
Revised: To do so, SynFlow computes the “synaptic strengths” of each connection and prunes those weights with the lowest synaptic strengths (see Algorithm 2 below for the details on computing the synaptic strengths).

Original: Importantly, SynFlow prunes iteratively.
Revised: Importantly, SynFlow prunes iteratively.

Original: It prunes a small number of weights, recalculates the synaptic strengths once those weights have been ﬁxed to zero, and then prunes again.
Revised: It prunes a small number of weights, recalculates the synaptic strengths once those weights have been ﬁxed to zero, and then prunes again.

Original: To prune to sparsity s , SynFlow iteratively prunes from sparsity s n − 1 100 to sparsity s n 100 for n ∈ { 1 , . .
Revised: To prune to sparsity s , SynFlow iteratively prunes from sparsity s n − 1 100 to sparsity s n 100 for n ∈ { 1 , . .

Original: After pruning, SynFlow trains the network normally using the standard hyperparameters.
Revised: After pruning, SynFlow trains the network normally using the standard hyperparameters.

Original: SynFlow computes the synaptic strengths as follows:
Revised: SynFlow computes the synaptic strengths as follows:

Original: Algorithm {+
+}
Revised: Algorithm {+
+}

Revised: Computing the strengths weights W .

Original: 1: Replace all weights w ∈ W with their magnitudes | w | .
Revised: 1: Replace all weights w ∈ W with their magnitudes | w | .

Original: Forward propagate an input of all 1’s 3: Take the sum of the logits R . 

Revised: Forward propagate an input of all 1’s 3: Take the sum of the logits R .

Original: : The synaptic strength
Revised: : The synaptic strength

Original: Note that this algorithm leads to exploding activations on deeper networks, so we do not vary network depths in any of our experiments involving SynFlow.
Revised: Note that this algorithm leads to exploding activations on deeper networks, so we do not vary network depths in any of our experiments involving SynFlow.

Original: B E XPERIMENTAL D ETAILS B. R ES N ETS
Revised: B E XPERIMENTAL D ETAILS B. R ES N ETS

Original: We study the residual networks (ResNets) designed by He et al.
Revised: We study the residual networks (ResNets) designed by He et al.

Original: (2016) for CIFAR-10 and ImageNet.
Revised: (2016) for CIFAR-10 and ImageNet.

Original: ResNets for CIFAR-10 are composed of an initial convolutional layer, three sets of B residual blocks (each with two convolutional layers and a skip connection), and a linear output layer.
Revised: ResNets for CIFAR-10 are composed of an initial convolutional layer, three sets of B residual blocks (each with two convolutional layers and a skip connection), and a linear output layer.

Original: The sets of blocks have 16, 32, and 64 convolutional channels, respectively.
Revised: The sets of blocks have 16, 32, and 64 convolutional channels, respectively.

Original: ResNets for ImageNet are composed of an initial convolutional layer, a max-pooling layer, four sets of residual blocks (each with three convolutional layers and a skip connection), and a linear output layer.
Revised: ResNets for ImageNet are composed of an initial convolutional layer, a max-pooling layer, four sets of residual blocks (each with three convolutional layers and a skip connection), and a linear output layer.

Original: The sets of blocks have 64, 128, 256, and 512 convolutional channels, respectively.
Revised: The sets of blocks have 64, 128, 256, and 512 convolutional channels, respectively.

Original: We place batch normalization before the ReLU activations.
Revised: We place batch normalization before the ReLU activations.

Original: To vary the width of the networks, we multiply the number of convolutional channels by the width scaling factor w .
Revised: To vary the width of the networks, we multiply the number of convolutional channels by the width scaling factor w .

Original: To vary the depth of the CIFAR-10 ResNets, we vary the value of B .
Revised: To vary the depth of the CIFAR-10 ResNets, we vary the value of B .

Original: The depth l of the network is the total number of the layers in the network, not counting skip connections.
Revised: The depth l of the network is the total number of the layers in the network, not counting skip connections.

Original: B. VGG N ETWORKS
Revised: B. VGG N ETWORKS

Original: We study the VGG-16 variant of the VGG networks for CIFAR-10 as provided by the OpenLTH repository.
Revised: We study the VGG-16 variant of the VGG networks for CIFAR-10 as provided by the OpenLTH repository.

Original: The network is divided into ﬁve sections, each of which is followed by max pooling with kernel size 2 and stride 2.
Revised: The network is divided into ﬁve sections, each of which is followed by max pooling with kernel size 2 and stride 2.

Original: The sections contain 3x3 convolutional layers arranged as follows:
Revised: The sections contain 3x3 convolutional layers arranged as follows:

Original: SectionWidthLayers  2 2 3 

Revised: SectionWidthLayers  2 2 3

Original: The network has ReLU activations and batch normalization before each activation.
Revised: The network has ReLU activations and batch normalization before each activation.

Original: To vary the width of VGG-16, we multiply each of the per-segment widths by the width scaling factor w .
Revised: To vary the width of VGG-16, we multiply each of the per-segment widths by the width scaling factor w .

Original: When pruning VGG, we consider the following conﬁgurations:
Revised: When pruning VGG, we consider the following conﬁgurations:

Original: [Table]
Revised: [Table]

Original: B. T RAINING H YPERPARAMETERS
Revised: B. T RAINING H YPERPARAMETERS

Original: We train CIFAR-10 ResNets and VGG-16 for 160 epochs with a batch size of 128.
Revised: We train CIFAR-10 ResNets and VGG-16 for 160 epochs with a batch size of 128.

Original: The initial learning rate is 0.1, and it drops by an order of magnitude at epochs 80 and 120.
Revised: The initial learning rate is 0.1, and it drops by an order of magnitude at epochs 80 and 120.

Original: We optimize using SGD with momentum (0.9).
Revised: We optimize using SGD with momentum (0.9).

Original: We initialize with He uniform initialization.
Revised: We initialize with He uniform initialization.

Original: Data is augmented by normalizing, randomly ﬂipping left and right, and randomly shifting by up to four pixels in any direction (and cropping afterwards).
Revised: Data is augmented by normalizing, randomly ﬂipping left and right, and randomly shifting by up to four pixels in any direction (and cropping afterwards).

Original: All CIFAR-10 networks are trained on GPUs.
Revised: All CIFAR-10 networks are trained on GPUs.

Original: We train ImageNet ResNets for 90 epochs with a batch size of 1024.
Revised: We train ImageNet ResNets for 90 epochs with a batch size of 1024.

Original: The initial learning rate is 0.4, and it drops by an order of magnitude at epochs 30, 60, and 80.
Revised: The initial learning rate is 0.4, and it drops by an order of magnitude at epochs 30, 60, and 80.

Original: We perform linear learning rate warmup from 0 to 0.4 over the ﬁrst 5 epochs.
Revised: We perform linear learning rate warmup from 0 to 0.4 over the ﬁrst 5 epochs.

Original: We optimize using SGD with momentum (0.9).
Revised: We optimize using SGD with momentum (0.9).

Original: We initialize with He uniform initialization.
Revised: We initialize with He uniform initialization.

Original: Data is augmented by normalizing, randomly ﬂipping left and right, selecting a random aspect ratio between 0.8 and 1.25, selecting a random scaling factor between 0.1 and 1.0, and cropping accordingly.
Revised: Data is augmented by normalizing, randomly ﬂipping left and right, selecting a random aspect ratio between 0.8 and 1.25, selecting a random scaling factor between 0.1 and 1.0, and cropping accordingly.

Original: All ImageNet networks are trained on GPUs.
Revised: All ImageNet networks are trained on GPUs.

Original: C F ULL D ATA FOR K EY O BSERVATIONS IN S ECTION
Revised: C F ULL D ATA FOR K EY O BSERVATIONS IN S ECTION

Original: In this appendix, we show that our observations from Section 3 hold when varying all dimensions (depth, width, and dataset size) on both the CIFAR-10 and ImageNet ResNets for IMP.
Revised: In this appendix, we show that our observations from Section 3 hold when varying all dimensions (depth, width, and dataset size) on both the CIFAR-10 and ImageNet ResNets for IMP.

Original: Figureshows the error versus density when changing width (left) depth (center) and data (right).
Revised: Figureshows the error versus density when changing width (left) depth (center) and data (right).

Original: In Figure 8, we similarly show the dependency of the error on density for Imagenet when varying width (left) and dataset size (right).
Revised: In Figure 9, we similarly show the dependency of the error on density for Imagenet when varying width (left) and dataset size (right).

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we observe that all curves have a similar slope in the power-law region.
Revised: we observe that all curves have a similar slope in the power-law region.

Original: In Equation 1, this implies that while γ is allowed to vary with l , w and n
Revised: In Equation 1, this implies that while γ is allowed to vary with l , w and n

Original: [Figure]
Revised: [Figure]

Original: it is in practice approximately a constant.
Revised: it is in practice approximately a constant.

Original: Similarly, the high-error plateau (cid:15) ↑ is also shared across curves such that it too is approximately constant.
Revised: Similarly, the high-error plateau (cid:15) ↑ is also shared across curves such that it too is approximately constant.

Original: In contrast, the transition from high-error plateau to the power-law region is not constant as a function of density.
Revised: In contrast, the transition from high-error plateau to the power-law region is not constant as a function of density.

Original: Section 4 ﬁnds exactly this dependency of the transition parameter p
Revised: Section 4 ﬁnds exactly this dependency of the transition parameter p

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D P ARTIAL (P ROJECTIONS ) F IT R ESULTS FOR S ECTION
Revised: D P ARTIAL (P ROJECTIONS ) F IT R ESULTS FOR S ECTION

Original: In Section 4, we ﬁt the error jointly as a function of all dimensions showing that Equation 2 provides a good approximation to the error in practice.
Revised: In Section 4, we ﬁt the error jointly as a function of all dimensions showing that Equation 2 provides a good approximation to the error in practice.

Original: In this appendix, we consider important sub-cases, such as the case when one wishes to scale only one degree of freedom while pruning.
Revised: In this appendix, we consider important sub-cases, such as the case when one wishes to scale only one degree of freedom while pruning.

Original: This serves both a practical scenario, but also allows for a qualitative visualization of the ﬁt (and typical sources of error), which is otherwise difﬁcult to perform over all dimensions jointly.
Revised: This serves both a practical scenario, but also allows for a qualitative visualization of the ﬁt (and typical sources of error), which is otherwise difﬁcult to perform over all dimensions jointly.

Original: From a practical standpoint, in this case one need not estimate the parameters associated with the ﬁxed degree of freedom.
Revised: From a practical standpoint, in this case one need not estimate the parameters associated with the ﬁxed degree of freedom.

Original: Recall that, given the non-pruned network error (cid:15) np , all dependencies on the individual structural degrees of freedom l, w are captured by the invariant
Revised: Recall that, given the non-pruned network error (cid:15) np , all dependencies on the individual structural degrees of freedom l, w are captured by the invariant

Original: m ∗ (cid:44) l φ w ψ d .
Revised: m ∗ (cid:44) l φ w ψ d .

Original: This means that, if one wishes to estimate the error while pruning when holding width ﬁxed, we need not estimate ψ .
Revised: This means that, if one wishes to estimate the error while pruning when holding width ﬁxed, we need not estimate ψ .

Original: Similarly if depth is held constant, we need not estimate φ .
Revised: Similarly if depth is held constant, we need not estimate φ .

Original: Figure 9 shows these partial ﬁts.
Revised: Figure 10 shows these partial ﬁts.

Original: Shown from left to right are the ﬁts done while pruning and varying width, depth and data respectively.
Revised: Shown from left to right are the ﬁts done while pruning and varying width, depth and data respectively.

Original: Correspondingly, these ﬁts omit separately ψ or φ or omit both when depth nor width are scaled.
Revised: Correspondingly, these ﬁts omit separately ψ or φ or omit both when depth nor width are scaled.

Revised: The ﬁts were performed with all available density points for each dimension.

Revised: For CIFAR-10: 7 widths, 224 points for the width partial ﬁt; 7 dataset fractions,points for the data partial ﬁt; 4 depths, 164 points for the depth partial ﬁt.

Revised: For ImageNet: 5 widths,points for the width partial ﬁt; 3 dataset fractions, 86 points for the data partial ﬁt.

Original: This exercise, apart from its practical implications, highlights the fact that there are in effect two groups of parameters comprising the estimation.
Revised: This exercise, apart from its practical implications, highlights the fact that there are in effect two groups of parameters comprising the estimation.

Original: The ﬁrst are the parameters (cid:15) ↑ , γ and p (cid:48) which control the dependency as a function of density (or more generally, as a function of the invariant).
Revised: The ﬁrst are the parameters (cid:15) ↑ , γ and p (cid:48) which control the dependency as a function of density (or more generally, as a function of the invariant).

Original: The second are φ and ψ which are properties of the architectural degrees of freedom captured by the invariant.
Revised: The second are φ and ψ which are properties of the architectural degrees of freedom captured by the invariant.

Original: Moreover, within the ﬁrst group of parameters (cid:15) ↑ , γ , can be isolated and found from a single pruning curve, as they are not a function of l, w, n .
Revised: Moreover, within the ﬁrst group of parameters (cid:15) ↑ , γ , can be isolated and found from a single pruning curve, as they are not a function of l, w, n .

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: l o
Revised: l o

Original: [Table]
Revised: [Figure]

Revised: t o p

Original: [Figure]
Revised: [Figure]

Revised: rr )

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] e s t

Revised: i

Original: [Figure]
Revised: [Figure]

Original: l o g 1( t o pe rr ) error vs m* fitw
Revised: a t e d e rr o

Original: [Figure]
Revised: [Figure]

Original: w
Revised: [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: E A DDITIONAL P RUNING A LGORITHMS AND A RCHITECTURES
Revised: E A DDITIONAL P RUNING A LGORITHMS AND A RCHITECTURES

Original: In this appendix, we show that our functional form applies to both an additional network architecture (VGG-16 on CIFAR-10) and an additional pruning algorithm (SynFlow).
Revised: In this appendix, we show that our functional form applies to both an additional network architecture (VGG-16 on CIFAR-10) and an additional pruning algorithm (SynFlow).

Original: We add these additional comparisons in the following Figures:
Revised: We add these additional comparisons in the following Figures:

Original: • Figure 10: VGG-16 on CIFAR-10 with IMP as width varies.
Revised: • Figure 11: VGG-16 on CIFAR-10 with IMP as width varies.

Original: µ < 3% , σ < 7% .
Revised: µ < 3% , σ < 7% .

Original: Notably, the measurement error in this case is large ( σ ∼ 12% ), dominating (over the approximation error)
Revised: Notably, the measurement error in this case is large ( σ ∼ 12% ), dominating (over the approximation error)

Original: the total ﬁt error.
Revised: the total ﬁt error.

Original: The ﬁt averages out some of this error, resulting in a ﬁt error which is lower than the measurement error.
Revised: The ﬁt averages out some of this error, resulting in a ﬁt error which is lower than the measurement error.

Original: • Figure 11: ResNet-20 on CIFAR-10 with SynFlow as width varies.
Revised: • Figure 12: ResNet-20 on CIFAR-10 with SynFlow as width varies.

Original: µ < 1% , σ < 4% .
Revised: µ < 1% , σ < 4% .

Original: • Figure 12: VGG-16 on CIFAR-10 with SynFlow as width varies.
Revised: • Figure 13: VGG-16 on CIFAR-10 with SynFlow as width varies.

Original: µ < 1% , σ < 4% .
Revised: µ < 1% , σ < 4% .

Original: Note that SynFlow suffers from exploding activations on deeper networks, so we do not vary ResNet depth in any of our SynFlow experiments.
Revised: Note that SynFlow suffers from exploding activations on deeper networks, so we do not vary ResNet depth in any of our SynFlow experiments.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: l o
Revised: o

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: l o
Revised: l o

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t o pe rr )
Revised: t o pe rr )

Original: [Figure]
Revised: [Figure]

Original: e s t i m a t e d e rr o r fit :-0.8% :3.8% 0.
Revised: e s t i m a t e d e rr o r fit :-0.8% :3.8% 0.

Original: [Figure]
Revised: [Figure]

Original: (mean) 0.
Revised: (mean) 0.

Original: a c t u a l e rr o r ( n o m e a n ) baseline :0.1% :3.1%
Revised: a c t u a l e rr o r ( n o m e a n ) baseline :0.1% :3.1%

Original: F I NTERPOLATION - FIT SENSITIVITY ANALYSIS

Original: In Section 4 and Appendix E, we showed that our scaling law was accurate when we ﬁt it on all of the available data.

Original: Now that we possess the functional form and know that it can accurately model the behavior of IMP and SynFlow pruning, we study the amount of data necessary to obtain a stable, 11 accurate ﬁt.

Original: This question is especially relevant when the functional form is applied to new settings—new networks, datasets, or pruning algorithms—and we must collect new data to do so.

Original: The functional form has only ﬁve parameters, suggesting that few experiments will be necessary.

Original: Experiments.

Original: To answer this question, we randomly sample varying numbers of points, ﬁt the scaling law to those points, and evaluate the quality of the ﬁt over all points.

Original: We sample these points in two ways.

Original: Experiment 1.

Original: Randomly sample T network conﬁgurations ( w, l, n, d ) .

Original: This experiment captures the use case of algorithms such as SynFlow, where obtaining data at any density d relies only on possessing the unpruned network, not other densities d (cid:48) > d .

Original: Experiment 2.

Original: Randomly sample T network conﬁgurations ( w, l, n ) and include all densities d for each conﬁguration.

Original: This experiment captures the use case of algorithms such as IMP, where obtaining data at density d requires obtaining all densities d (cid:48) > d .

Original: As such, we anticipate that data will be obtained by iteratively pruning a small number of conﬁgurations ( w, l, n ) to low density.

Original: Results.

Original: We perform each experiment for many different values of T on the CIFAR-10 ResNets pruned with IMP.

Original: We repeat the experiment at each value of T 30 times and report the mean and standard deviation of µ and σ for the ﬁt.

Original: Experiments 1 and 2 respectively appear in Figures 13 and 14.

Original: The shaded areas represent one standard deviation from the mean in each direction.

Original: Experiment 1.

Original: When just 40 conﬁgurations of ( w, l, d, n ) are available, the standard deviation on both µ and σ is just one percentage point.

Original: Experiment 2.

Original: When just 15 random conﬁgurations of ( w, l, n ) are available at all densities, we similarly achieve standard deviation below 1%.

Original: In both cases, as the number of networks increases, the standard deviation decreases further.

Original: Discussion.

Original: The above results show that, now that our scaling law is known, it is possible to obtain an accurate (and stable) estimation using far less data than we used to evaluate the quality of the ﬁt in the main body of the paper.

Original: Importantly, the experiments we perform in this section are particularly naive.

Original: We make no effort to ensure that the conﬁgurations we select represent a diverse range of widths, depths, dataset sizes, and densities.

Original: By selecting these conﬁgurations in a strategic way, we believe it would be possible to further reduce the number of conﬁgurations necessary to obtain a similarly accurate ﬁt.

Original: [Figure]

Original: [Figure]

Original: G T OWARDS E XTRAPOLATION
Revised: F T OWARDS E XTRAPOLATION

Original: Background.
Revised: Background.

Original: In the main body, we showed that our scaling law accurately ﬁts the error of pruned neural networks.
Revised: In the main body, we showed that our scaling law accurately ﬁts the error of pruned neural networks.

Original: As, such it has predictive power, allowing us to reason in a principled manner about pruning trade-offs.
Revised: As, such it has predictive power, allowing us to reason in a principled manner about pruning trade-offs.

Original: Similarly, it allows to make predictions about what would happen at larger model and data scales than explored here.
Revised: Similarly, it allows to make predictions about what would happen at larger model and data scales than explored here.

Original: Importantly, only a few experiments need be performed to ﬁnd the coefﬁcients for the scaling law (see Appendix F).
Revised: Importantly, only a few experiments need be performed to ﬁnd the coefﬁcients for the scaling law (see Appendix 5).

Original: However, we could ask, how accurately can we estimate the scaling law parameters from even smaller scales?
Revised: However, we could ask, how accurately can we estimate the scaling law parameters from even smaller scales?

Original: That is, is it possible to ﬁt our scaling law to data from networks with deliberately smaller depths, widths, and dataset sizes and accurately predict the error of larger-scale models?
Revised: That is, is it possible to ﬁt our scaling law to data from networks with deliberately smaller depths, widths, and dataset sizes and accurately predict the error of larger-scale models?

Original: If so, we could make informed decisions about pruning large-scale models through small-scale experiments alone, saving the costs associated with large scale training and pruning.
Revised: If so, we could make informed decisions about pruning large-scale models through small-scale experiments alone, saving the costs associated with large scale training and pruning.

Original: Outside the context of pruning, the scaling laws of Rosenfeld et al.
Revised: Outside the context of pruning, the scaling laws of Rosenfeld et al.

Original: (2020) (for both language models and image classiﬁcation) and Kaplan et al.
Revised: (2020) (for both language models and image classiﬁcation) and Kaplan et al.

Original: (2020) (for predicting the expected performance of GPT- (Brown et al., 2020) at very large scale) have been shown to extrapolate successfully in this manner.
Revised: (2020) (for predicting the expected performance of GPT- (Brown et al., 2020) at very large scale) have been shown to extrapolate successfully in this manner.

Original: Results on CIFAR-10.
Revised: Results on CIFAR-10.

Original: In Figure 15, we show the result of extrapolating from small-scale networks on CIFAR-10 ( w = 1 8 , 1 4 ; l = 14 , 20 ) to all widths and depths on CIFAR-10.
Revised: In Figure 14, we show the result of extrapolating from small-scale networks on CIFAR-10 ( w = 1 8 , 1 4 ; l = 14 , 20 ) to all widths and depths on CIFAR-10.

Original: Extrapolation prediction is still accurate: µ < 7% , σ < 6% (vs. µ < 1% , σ < 6% in the main body).
Revised: Extrapolation prediction is still accurate: µ < 7% , σ < 6% (vs. µ < 1% , σ < 6% in the main body).

Original: Future work.
Revised: Future work.

Original: However, extrapolation is particularly sensitive to systemic errors.
Revised: However, extrapolation is particularly sensitive to systemic errors.

Original: Speciﬁcally, the transitions and the error dips can lead to large deviations when extrapolating.
Revised: Speciﬁcally, the transitions and the error dips can lead to large deviations when extrapolating.

Original: For ImageNet, the error dips (especially on small dataset sizes) are especially pronounced, preventing stable extrapolation.
Revised: For ImageNet, the error dips (especially on small dataset sizes) are especially pronounced, preventing stable extrapolation.

Original: In order to improve extrapolation performance, future work should explore the challenges we discuss in
Revised: In order to improve extrapolation performance, future work should explore the challenges we discuss in

Original: Section 5: approaches to either model or mitigate these dips and to improve the ﬁt of the transitions.
Revised: Section 6: approaches to either model or mitigate these dips and to improve the ﬁt of the transitions.

Original: 7 log10( #non-zero param ) 1.
Revised: 7 log10( #non-zero param ) 1.

Original: l o g( t o pe rr ) error vs #non-zero param extrapfit 0.
Revised: l o g( t o pe rr ) error vs #non-zero param extrapfit 0.

Original: 1.0 actual error (mean) 0.
Revised: 1.0 actual error (mean) 0.

Original: e s t
Revised: e s t

Original: i m a t e d e rr o r
Revised: i m a t e d e rr o r

Original: [Figure]
Revised: [Figure]

Original: H C OMPARISON OF P RUNING AND N ON - PRUNING S CALING L AWS
Revised: G C OMPARISON OF P RUNING AND N ON - PRUNING S CALING L AWS

Original: In this appendix, we contrast the behavior of the error when pruning with the behavior of the error in the non-pruning setting.
Revised: In this appendix, we contrast the behavior of the error when pruning with the behavior of the error in the non-pruning setting.

Original: Hestness et al.
Revised: Hestness et al.

Original: (2017) show the the error follows a saturating power-law form when scaling data (with both low and high-error plateaus) but does not model them.
Revised: (2017) show the the error follows a saturating power-law form when scaling data (with both low and high-error plateaus) but does not model them.

Original: Rosenfeld et al.
Revised: Rosenfeld et al.

Original: (2020) unify the dependency on data and model size while approximating the transitions between regions; they propose the following form:
Revised: (2020) unify the dependency on data and model size while approximating the transitions between regions; they propose the following form:

Original: [Equation]
Revised: [Equation]

Original: where m is the total number of parameters and n is the dataset size.
Revised: where m is the total number of parameters and n is the dataset size.

Original: a, b, α, β, c ∞ , and η are constants, and (cid:15) 0 plays the role of (cid:15) ↑ in our notation.
Revised: a, b, α, β, c ∞ , and η are constants, and (cid:15) 0 plays the role of (cid:15) ↑ in our notation.

Original: Rosenfeld et al.
Revised: Rosenfeld et al.

Original: model the upper transition—from power-law region to the high-error plateau—by a rational form in a fashion similar to the approach we take.
Revised: model the upper transition—from power-law region to the high-error plateau—by a rational form in a fashion similar to the approach we take.

Original: The key difference is that we consider a power of the polynomials in the numerator and denominator of the rational form, where in Eq.
Revised: The key difference is that we consider a power of the polynomials in the numerator and denominator of the rational form, where in Eq.

Original: 3 the power is hidden in the term ˜ (cid:15) .
Revised: 3 the power is hidden in the term ˜ (cid:15) .

Original: The biggest difference arises when considering the lower transition (between the low-error plateau and the power-law region).
Revised: The biggest difference arises when considering the lower transition (between the low-error plateau and the power-law region).

Original: This transition is captured by Eq. 3.
Revised: This transition is captured by Eq. 3.

Original: Considering either the width or depth degrees of freedom x ∈ { w, l } , Eq. 3 can be re-written as:
Revised: Considering either the width or depth degrees of freedom x ∈ { w, l } , Eq. 3 can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: Where b x and β x are constants and c x is a constant as a function of x (it is only a function of the data size n ).
Revised: Where b x and β x are constants and c x is a constant as a function of x (it is only a function of the data size n ).

Original: Figure 16 (right) shows the error versus depth for different dataset sizes.
Revised: Figure 15 (right) shows the error versus depth for different dataset sizes.

Original: In grey is the actual error, while in red is the best ﬁt when approximating the error by Eq. 5.
Revised: In grey is the actual error, while in red is the best ﬁt when approximating the error by Eq. 5.

Original: Qualitatively, one sees that the ﬁt using Eq.
Revised: Qualitatively, one sees that the ﬁt using Eq.

Original: 5 does indeed closely match the error in practice.
Revised: 5 does indeed closely match the error in practice.

Original: Recall that we are interested in comparing the errors as a function of the density.
Revised: Recall that we are interested in comparing the errors as a function of the density.

Original: A requirement from any functional form used to model the dependency on the density is to degenerate to the error of the non pruned model (cid:15) np at d = 1 .
Revised: A requirement from any functional form used to model the dependency on the density is to degenerate to the error of the non pruned model (cid:15) np at d = 1 .

Original: We adapt Eq. 5 by solving the relation between b x and c x meeting this constraint, to arrive at:
Revised: We adapt Eq. 5 by solving the relation between b x and c x meeting this constraint, to arrive at:

Original: [Equation]
Revised: [Equation]

Original: Contrast Eq.
Revised: Contrast Eq.

Original: 5 with the functional form we propose in Eq.
Revised: 5 with the functional form we propose in Eq.

Original: 1, re-written here for convenience:
Revised: 1, re-written here for convenience:

Original: [Equation]
Revised: [Equation]

Original: This can be simpliﬁed to capture only the lower transition—far enough from the upper transition ( d (cid:29) p )—to:
Revised: This can be simpliﬁed to capture only the lower transition—far enough from the upper transition ( d (cid:29) p )—to:

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) γ (8)
Revised: (cid:13)(cid:13)(cid:13) γ (8)

Original: Figure 16 (left) shows error versus density for different widths.
Revised: Figure 15 (left) shows error versus density for different widths.

Original: In blue is the ﬁt with Eq. 8 which follows closely the actual error (black) while in red is the ﬁt with Eq. 6 which deviates noticeably in comparison.
Revised: In blue is the ﬁt with Eq. 8 which follows closely the actual error (black) while in red is the ﬁt with Eq. 6 which deviates noticeably in comparison.

Original: [Figure]
Revised: [Figure]

Original: We have seen that in practice that the form of Eq.
Revised: We have seen that in practice that the form of Eq.

Original: 6 does not match well the pruning case, where the mismatch originates from lower transition shape.
Revised: 6 does not match well the pruning case, where the mismatch originates from lower transition shape.

Original: We have thus reached a phenomenological observation distinguishing the pruning and non-pruning forms; we leave the study of the origins of this phenomenon for future work.
Revised: We have thus reached a phenomenological observation distinguishing the pruning and non-pruning forms; we leave the study of the origins of this phenomenon for future work.

Original: I T HE EFFECT OF ERROR DIPS ON ESTIMATION BIAS
Revised: H T HE EFFECT OF ERROR DIPS ON ESTIMATION BIAS

Original: In this appendix, we consider the effect of the error dips on our estimator as discussed in Section 4.
Revised: In this appendix, we consider the effect of the error dips on our estimator as discussed in Section 4.

Original: As we mention in that section, when pruning a network, error often dips below (cid:15)
Revised: As we mention in that section, when pruning a network, error often dips below (cid:15)

Original: np during the low-error plateau.
Revised: np during the low-error plateau.

Original: Recall that we ﬁnd the parameters in our estimator (Equation 2) by minimizing the MSE of relative error δ .
Revised: Recall that we ﬁnd the parameters in our estimator (Equation 2) by minimizing the MSE of relative error δ .

Original: Our estimation has bias if E (ˆ (cid:15) − (cid:15) )
Revised: Our estimation has bias if E (ˆ (cid:15) − (cid:15) )

Original: (cid:54) = 0 where the expectation is over all model and data conﬁgurations.
Revised: (cid:54) = 0 where the expectation is over all model and data conﬁgurations.

Original: Equivalently, the relative bias is µ (cid:44) E δ = 

Revised: Equivalently, the relative bias is µ (cid:44) E δ =

Original: iff the estimator is unbiased.
Revised: iff the estimator is unbiased.

Original: The
Revised: The

Original: Estimator captured by the joint form in Equation 2 is a monotonically increasing function of the density.
Revised: Estimator captured by the joint form in Equation 2 is a monotonically increasing function of the density.

Original: It is also constrained such that at density d = 1 it is equal to the non-pruned error (cid:15) np .
Revised: It is also constrained such that at density d = 1 it is equal to the non-pruned error (cid:15) np .

Original: It thus, can not reduce The MSE to zero, as it can not decrease to match the actual error dips.
Revised: It thus, can not reduce The MSE to zero, as it can not decrease to match the actual error dips.

Original: This results in the bias of the relative error µ which in practice is ∼ 1% .
Revised: This results in the bias of the relative error µ which in practice is ∼ 1% .