U NIVERSALITY OF D EEP N EURAL N ETWORK L OTTERY T ICKETS : A R ENORMALIZATION G ROUP P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures.
While this has become of broad practical and theoretical interest, to date, there exists no detailed understanding of why winning ticket universality exists, or any way of knowing whether tickets can be transferred between two given tasks, without ﬁrst performing transfer experiments.
To address these outstanding open questions, we make use of renormalization group theory, one of the most successful tools in theoretical physics.
We ﬁnd that iterative magnitude pruning, the method used for discovering winning tickets, is a renormalization group scheme.
This opens the door to a wealth of existing numerical and theoretical tools, some of which we leverage here to examine winning ticket universality in large scale lottery ticket experiments, as well as sheds new light on the success iterative magnitude pruning has found generally in machine learning.
I NTRODUCTION
The lottery ticket hypothesis (LTH) for deep neural networks (DNNs) proposes that DNNs contain sparse subnetworks that can be trained in isolation and can reach performance that is equal to, or better than, that of the full DNN in the same number of training iterations (Frankle & Carbin, 2019;
Frankle et al., 2020a).
These subnetworks are called winning lottery tickets.
The LTH has provided paradigm shifting insight into the success of dense DNNs, suggesting a key role for the emergence of winning tickets with increasing size, reminiscent of the maxim “more is different” (Anderson, 1972).
In recent years, researchers have found an intriguing corollary: winning tickets found in the context of one task can be transferred to related tasks (Desai et al., 2019; Mehta, 2019; Morcos et al., 2019; Soelen & Sheppard, 2019; Chen et al., 2020a; Gohil et al., 2020; Chen et al., 2021a; Sabatelli et al., 2021), possibly even across different architectures (Chen et al., 2021c).
In addition to having applications of practical interest, these results imply that winning tickets can be used to study how tasks and architectures are “similar”.
However, to date, there exists no principled understanding of why winning tickets can be transferred between tasks, nor does there exist any way to know, without directly performing transfer experiments, which previously studied tasks a given winning ticket can be transferred to.
Furthermore, there is a general lack of theoretical work on iterative magnitude pruning (IMP) (Elesedy et al., 2021), the most commonly used method to ﬁnd winning tickets.
This is in striking analogy to the state of statistical physics in the early-to-mid–20 th century.
Empirical evidence suggested that disparate systems, governed by seemingly different underlying physics, exhibited the same , universal properties near their phase transitions.
While heuristic methods provided insight (Kadanoff, 1966), a full theory from ﬁrst principle on this universality was not realized until the development of the renormalization group (RG) (Wilson, 1971a;b; 1975).
RG theory has not only provided a framework for explaining universal behavior near phase transitions, but also a scheme for grouping systems by that behavior.
This classiﬁcation, introduced via the notion of universality classes, has allowed for a detailed understanding of materials (Tougaard, 1997; Winter & Mours, 1997; Durin & Zapperi, 2000; Bonilla et al., 2010), and provides immediate knowledge as to what other classiﬁed systems a given material behaves like.
The analogy between universality in RG and LTH theories becomes strengthened under the following consideration.
Recently, Rosenfeld et al.
(2020) found that when the density (i.e. the percentage of parameters remaining) of a DNN being pruned via IMP is in a certain range, d L < d < d C , the network’s error scales according to the following power-law,
[Equation]
Power-law scaling is well known to emerge in critical phenomena.
For instance, when the temperature of a classical spin system (e.g. two-dimensional Ising model) is near and below the critical temperature, t < t C , many observables (e.g. magnetization) exhibit power-law scaling,
[Equation]
These similarities hint at a more fundamental connection between the universality present in winning tickets and in physical systems near phase transitions (see Fig.
1 and Table 1).
Indeed, we show that IMP is an RG scheme (i.e. IMP satisﬁes the properties required to be considered an RG operator).
By analyzing winning ticket transfer experiments (Chen et al., 2021a), we ﬁnd that tasks that allow for successful transfer of winning tickets have similar behavior in their IMP ﬂow (i.e. the way in which the parameters of the DNN change after each iteration of IMP), supporting a key prediction of the theory.
We additionally leverage the RG framework to interpret recent results on transferring winning tickets across differing architectures (Chen et al., 2021c).
Our contributions are the following:
• A theoretical basis for understanding IMP and the universality of winning tickets; • Experimental support of the theory in large scale lottery ticket experiments, providing insight into previously known heuristic based results; • A new set of tools for studying the LTH and associated phenomena.
T HE R ENORMALIZATION G ROUP AND I TERATIVE M AGNITUDE P RUNING
The RG operator, R , can be viewed as a method for “coarse-graining”.
That is, each application of R replaces local degrees of freedom with a composite of their values 1 .
An example of this for the two-dimensional Ising model is shown in the left-hand column Fig.
1, where neighborhoods of four spins are replaced by their mode.
The formal way to study R is to consider its action on a given Hamiltonian (i.e. energy function).
For classical spin systems (e.g. two-dimensional Ising model), H has the general form
[Equation]
where the s i are the spins of the system (e.g. s i ∈ {− 1 , +1 } ), (cid:104)· , ·(cid:105) represents sites on the lattice that are nearest neighbors, and the k i are the strengths of the different coupling constants (e.g. k 2 is the strength of the nearest neighbor coupling).
Due to the fact that coarse-graining amalgamates spins, the spin system resulting from an application of R can be viewed as equivalent to the original, but with a new set of coupling constants.
Therefore, the coarse-grained system has a different Hamiltonian, which is given by
[Equation]
where s (cid:48) is the new set of spins and k (cid:48) is the new set of couplings determined by the operator T : R K → R K .
Here K is the maximum number of couplings considered, usually introduced to keep the considered operators ﬁnite.
R can be applied iteratively, deﬁning a ﬂow through the function space of Hamiltonians via Eq.
4, with an associated ﬂow in the space of coupling constants.
The latter is commonly referred to as the
RG ﬂow.
While R is often a complicated, non-linear function, T can be linearized near ﬁxed points (Goldenfeld, 2005).
The ﬂow will grow in the direction of the eigenvectors v r with eigenvalues λ r > 1 and will shrink along the direction of the eigenvectors v i with λ
i < 1 .
These eigenvectors, which are linear combinations of coupling constants are called relevant and irrelevant , respectively.
A schematic of these, and the general RG ﬂow, is presented in Fig.
By examining the effect of applying R to a given system, and its ﬂow in the space of coupling constants, it is possible to ﬁnd which components of the system are necessary for certain macroscopic behavior (i.e those that are relevant), and which are not.
Systems belonging to a particular universality class have the same relevant directions.
Interestingly, many of the properties discussed above can be analogously found for IMP.
To see this, consider a DNN with loss function L ( a , θ ) , where a is the unit activations and θ is the DNN parameters. Let I represent a single application of the IMP process.
The DNN pruned via an application of I is given by the relation
[Equation]
where the new set of parameters, θ (cid:48) , are given by an operator T .
This new set of parameters leads to a new, coarse-grained set of activations, a ’.
The similarities between Eqs. 4 and 5 are striking.
We note here that, in the case of IMP, T is the composition of a masking operator, M , and a reﬁning operator F .
That is, T = F ◦ M : R N → R N , where N is the number of parameters of the [Table] DNN.
M is deﬁned via the pruning procedure implemented (e.g. magnitude pruning).
While we are considering IMP here, because of its connection to the Lottery Ticket Hypothesis (Frankle &
Carbin, 2019; Frankle et al., 2020a), there are many other pruning procedures [e.g. Hessian based pruning (LeCun et al., 1989)], all of which have their own M .
Similarly, F is deﬁned via the reﬁnement procedure used, making it dependent on the choice of optimizer and whether or not the DNN parameters are left alone or “rewound” to their value at a previous point during training (Frankle et al., 2020a; Renda et al., 2020).
For n iterations of I , the resulting network is given by
[Equation]
This deﬁnes a trajectory in parameter space, θ (0) → θ [Equation]
→ ... → θ ( n − 1) , which we will refer to as the IMP ﬂow .
A schematic illustration of what the IMP ﬂow might look like is given in Fig.(the axes of which will be explained in Sec. 3.1).
Just as in the case of RG theory, the directions of the IMP ﬂow are determined by the eigenvectors of T , which grow or shrink exponentially by the magnitude of their associated eigenvalues.
2. C ONNECTION B ETWEEN THE RG AND S TANDARD LTH F RAMEWORKS
Before we formally show that IMP is an RG scheme, we discuss how the standard theory that has emerged from studying the LTH is related to RG theory.
In the familiar picture, the success of winning tickets is attributed to the surviving sparse DNN being able to ﬁnd a “good” local minimum in the loss landscape (Frankle & Carbin, 2019; Frankle et al., 2020a).
This may be possible because the training of DNNs via stochastic gradient descent (SGD) appears to be rapidly conﬁned to a low-dimensional subspace (Gur-Ari et al., 2018), implying that
DNNs only “feels” changes to a small number of parameters during much of training.
Parameters that are not in this low-dimensional subspace can, therefore, be removed with minimal impact.
If a sparse DNN is initialized in this subspace (as late rewinding aims to do), then it may be possible for training to ﬁnd the same, or related, local minima as the full DNN (Evci et al., 2020; Maene et al., 2021).
In this way, the parameters that do not lie in the low-dimensional subspace are irrelevant, and repeated applications of IMP are expected to remove them.
On the other hand, the relevant parameters are those that span this subspace, and their removal changes the local minimum that the sparse DNN converges to.
Therefore, certain observable functions of the DNN, such as error, are expected to only be sensitive to the relevant, but not irrelevant, parameters.
If two models share the same low-dimensional subspace, then they should be able to transfer winning tickets.
Note that this transferability is highly non-trivial.
Indeed, it was only with the development of the LTH, and subsequent experimental work, that this was considered to be a possibility (Frankle & Carbin, 2019; Mehta, 2019; Morcos et al., 2019).
It is, in general, difﬁcult to ﬁnd these subspaces and accurately compare them across experiments.
RG theory provides us with tools for ﬁnding relevant and irrelevant directions, which do not rely on explicitly constructing the underlying subspace.
In particular, RG theory says that if two models have the same eigenvectors of T with eigenvalues > 1 , then they have the same relevant parameters.
This means that, once the eigenvalues are computed, we will know whether winning tickets can be transferred between two models, without having to run any additional experiments.
Note that the minimal sparsity a winning ticket can have, and still successfully be transferred, is not identiﬁed.
2. IMP AS AN RG S CHEME
To make the connection between IMP and RG more precise, we show that IMP ﬁts the deﬁnition of an RG scheme.
To do this, we consider the projection operator, P , that is associated with the
RG operator.
In the case of classical spin systems, the projection operator maps the spins, s i , to a coarse-grained spin system, s (cid:48) I , such that it satisﬁes
[Equation]
where Tr { s i } is the trace operator over the values that the s i can take (e.g. ± 1 ) (Goldenfeld, 2005).
For the two-dimensional Ising model, it is standard to take
[Equation]
where δ is the Kronecker delta function and sign ( · ) is +1 if the argument is positive and − 1 if it is negative (Goldenfeld, 2005).
Eq.
8 formally deﬁnes the mapping from s i → s (cid:48) I as
[Equation]
The projection operator is not unique, but must satisfy the following three properties:
P ( s i , s (cid:48) I ) ≥ 0 ; 2.
P ( s i , s (cid:48) I ) respects the symmetry of the system; 3.
(cid:88) { s (cid:48) I } P ( s i , s (cid:48) I )
= 1 .
To ﬁnd a projection operator associated with I , we start by ﬁnding a mapping between the activations of all the units, a , before and after an application of IMP. This is because a , and not θ , is the analogous quantity to s (Table 1 and Fig. 1).
Without loss of generality, we can consider the activation of unit j in layer i as being deﬁned by
[Equation]
where h is the activation function (e.g. ReLu, sigmoid) and the g k are functions that determine how the different parameters and activations of other units affect a ( i ) j .
For instance, in a feedforward
DNN, the impact of the bias of unit j in layer
i is given by g
= θ ( i ) j , and the weighted input from the previous layer is given by g 1 = (cid:80)
[Equation] i
[Equation]
Here N ( i − 1) is the number of units in layer i − 1 .
As discussed above, IMP changes the parameters θ by the operator T , which is deﬁned by a composition of M and F .
Therefore, the activation of unit j in layer i , after applying I , is given by
[Equation]
and thus, the projection operator associated with I is
[Equation]
We can easily verify that the projection operator deﬁned by Eq.
12 satisﬁes all three properties needed for an RG projection operator.
First, property #1 is satisﬁed, as Eq.
12 is a product of Kronecker delta functions.
Property #2 is satisﬁed, as IMP only removes parameters, so the form of a ( i ) j given in Eq.
11 and, thus, the loss function, will remain intact until layer collapse (i.e. when all the weights from one layer to another are removed).
Finally, for property #3 to be satisﬁed, we can ﬁx the test and training sample ordering for each epoch (as is done when the random seed is ﬁxed).
In such a case, both the mask and reﬁning operators in Eq.
12 are deterministic and P (cid:16) a ( i ) j , a (cid:48)
j ( i ) (cid:17) will be unique.
Having shown that these three properties are satisﬁed by the projection operator associated with IMP, we have found that I meets the criteria for being an RG operator, making it an RG scheme.
To the best of our knowledge, this has not been previously identiﬁed and provides new insight into why IMP has found success in discovering winning tickets (Frankle & Carbin, 2019; Frankle et al., 2020a), as well as has found general success in the study of DNNs (Frankle et al., 2020b).
R ESULTS 3. U NIVERSALITY IN THE IMP F LOW
We start by examining the IMP ﬂow of computer vision models.
The goal of this analysis is to examine DNNs that are known to allow for the successful transfer of winning tickets, and determine whether they have the same relevant and irrelevant parameters.
If they do, this would conﬁrm a major prediction of the theory developed in Sec. 2.
For spin systems in statistical physics, the RG ﬂow is studied in the space of coupling constants (Fig. 1).
Each coupling constant is usually assumed to be the same for all spins, greatly reducing the dimensionality of the underlying space.
While most DNNs do not set all parameters of a given type to the same value, we can none-the-less estimate the “inﬂuence” the parameters of a given layer or residual block have on the full DNN, by considering the functions
[Equation]
(13) where M i ( n ) describes the percentage of all non-zero parameters that are in layer or residual block i after IMP iteration n .
The numerator sums over all N ( i ) of the elements of the pruning mask restricted to layer or residual block i , m ( i ) .
Each m ( i ) j ∈ { 0 , 1 } .
The denominator sums the entire pruning mask, making it equivalent to the number of non-pruned parameters in the DNN.
Given the analogous role the M i play to the coupling constants of spin systems, we expect that they may be eigenfunctions of I , and, therefore, may encode the relevant and irrelevant components of a given
DNN. Because ResNet-50 has four residual blocks, IMP induces a four-dimensional ﬂow in the space of M i .
To visualize this trajectory, we examined two-dimensional slices.
Examples of these are plotted in Fig.
2 for ResNet-50, pre-trained on ImageNet via simCLR self-supervised training (Chen et al., 2020b). The ﬂow for three different tasks are shown: ImageNet, CIFAR-10, and CIFAR-100.
This data comes from experiments performed by Chen et al.
(2021a).
If the M i are eigenfunctions of the IMP operator, then they will scale exponentially with respect to the number of iterations I has been applied.
That is, M i ( n + 1) = I M i ( n )
= λ i M i ( n ) = λ n +1 i M (0) , where λ i is the eigenvalue governing its exponential growth/decay.
We can ﬁnd λ i by the simple inversion, λ i =
M i ( n + 1) /M
i ( n ) .
As long as M i ( n ) (cid:54) = 0 , this is well deﬁned [Table] and we can compute λ i by doing this for each pair of data points.
Performing this on our dataset yields estimates of eigenvalues with standard deviations < 5% (Table 2), suggesting that it is indeed reasonable to consider the M i as being eigenfunctions of I . Additionally, we ﬁnd that ImageNet, CIFAR-10, and CIFAR-100 have similar distributions in eigenvalues (Table 2).
In particular, all three identify the ﬁrst two residual blocks as being relevant and identify the last block as being marginal or irrelevant. From the theory developed in Sec. 2, these results imply that all three models are in similar universality classes and that winning tickets can be transferred between them.
Chen et al. (2021a) indeed conﬁrmed this, ﬁnding that winning tickets found on ImageNet could be successfully transferred to both CIFAR-10 and CIFAR-100.
This universality in IMP eigenvalues may be due to the fact that the ﬁrst few residual blocks learn low level statistics of the data (Neyshabur et al., 2020), which scale in speciﬁc, non-trivial ways for natural images (Ruderman & Bialek, 1994; Saremi & Sejnowski, 2013).
The sensitivity to the ﬁrst two residual blocks, which are the furthest from the output layer, is in contrast to standard spin systems.
In the case of the usual two-dimensional Ising model, the RG removes long-range interactions (i.e. couplings between next-nearest neighbors, next next-nearest neighbors, etc.), which are typically assumed, for physical reasons, to be weak.
Systems with relevant long-range interactions have been found to have unique properties, such as the existence of a phase transition in one-dimension (Dyson, 1969), which is known not to happen for the standard Ising model.
The importance of these long-range interactions may therefore be another interpretation for the beneﬁt of pre-training and normalized initialization (e.g. He initialization) for convolutional DNNs, as they are presumably part of what sets the ResNet-50 to be in this “long-range interaction” regime. However, we emphasize that the universal IMP eigenvalues (Table 2) are not merely a trivial reﬂection of the pre-training and normalized initialization.
In particular, the speciﬁc task matters, as each model starts from a slightly different point in the four-dimensional space of M i
[ImageNet: (0.0079, 0.0557, 0.3055, 0.6309); CIFAR-10: (0.0075, 0.0556, 0.3176, 0.6193); CIFAR-100: (0.0070, 0.0519, 0.3004, 0.6407)], implying that each task arrives at distinct distributions of parameters, despite starting from the same initial set of parameters. While pre-training and normalized initialization likely constrain the degree of these differences, it is conceivable that there are tasks, which would sufﬁciently re-organize the parameter distributions, such that earlier blocks could become irrelevant and/or later blocks could become relevant. In such cases, RG theory predicts that these tasks would not admit the transfer of winning tickets. Future work should attempt to identify such tasks.
Similar results as those reported in Table 2 were found for ResNet-50, pre-trained on ImageNet using supervised learning (Huh et al., 2016)
(Apppendix A), which was also previously shown to have successful winning ticket transfer between tasks (Chen et al., 2021a). While the theory developed in Sec. 2 made no assumption on architecture, we also computed IMP eigenvalues on large scale lottery ticket experiments using VGG-16 and BERT models, ﬁnding again that the M i could reasonably be considered eigenfunctions of I (Appendix B). This further supports our claim that the RG framework is general and can be broadly used to study winning ticket transferability. [Table]
E LASTIC L OTTERY T ICKET H YPOTHESIS
Recent work on the “Elastic Lottery Ticket Hypothesis” (E-LTH) has extended the notion of tranferability by ﬁnding it possible to transform winning tickets found on one DNN, to another with a different architecture (Chen et al., 2021c). In particular, it was found that, for architectures in the same family (e
[Table] g. ResNets) trained on the same task (e [Table] g. CIFAR-10), it was possible to either squeeze (by removing residual blocks) or stretch (by replicating residual blocks) winning tickets so that they could be transferred.
Three results from this study of the E-LTH are especially noteworthy:
The smallest ResNet considered (ResNet-14) had the weakest transferability properties; 2.
The more unique residual blocks replicated, the better the performance; 3.
Removing the later residual blocks, in the case of shrinking, or replicating the earlier residual blocks, in the case of stretching, led to the best results.
While preliminary hypotheses on the origin of these results were developed by Chen et al.
(2021c)
[Table]
involving dynamical systems and unrolled estimation, detailed theoretical understanding was left to future work.
Given that the RG framework developed above led to insight into the nature of the transfer of winning tickets in the case of different tasks, we wondered whether it could similarly help to explain the nature of transfer in the case of different architectures.
As discussed in Sec. 2, two tasks that have the same IMP eigenvalues can transfer winning tickets.
Therefore, we expect that transforming a given winning ticket from one architecture (source) to another (target) will be successful if its IMP ﬂow matches that of the target.
In particular, the success (or lack-thereof) should rely on whether the number of relevant, irrelevant, and marginal residual blocks in the target architecture is matched by the transformed source ticket.
To test this, we computed the eigenvalues associated with the eigenfunctions M i
(Eq. 13), for each “normal” residual block of ResNet-14, ResNet-20, ResNet-32, ResNet-44, and ResNet-56, trained on CIFAR-10 [data from Chen et al.
(2021c)].
As discussed in the original paper, the downsampling residual block in each stage was not transformed between winning tickets, so they were not considered.
The eigenvalues for the residual blocks in the ﬁrst stage are presented in Table 3.
We ﬁnd that, while the smaller ResNets (ResNet-14 and ResNet-20) have only relevant residual blocks, the larger ResNets have at least one non-relevant block.
This offers a straightforward explanation as to why ResNet-14 had particularly poor transferability: using it as the source ticket necessarily leads to the transformed ticket having only relevant residual blocks, which does not match the target tickets’ structure.
Having only relevant residual blocks may make the ﬁrst stage too sensitive as the depth of the ResNet is increased, leading to poorer performance.
This is likely related to the idea that smaller ResNet models are not sufﬁciently expressive, but has the beneﬁt of being theoretically backed and quantitative.
Relatedly, we ﬁnd that replicating few unique residual blocks (i.e. replicating only block 1 or blocks 1 and 2) can lead to an over representation of either relevant or non-relevant blocks.
This mismatch is predicted by the theory to lead to worse transferability.
Finally, we ﬁnd that most of the non-relevant residual blocks are in the early part of the ﬁrst stage.
Therefore, removing the ﬁrst several blocks when shrinking a winning ticket (e.g. dropping the ﬁrst four blocks to go from ResNet-56 to ResNet-32), or replicating the last several blocks when stretching a winning ticket (e.g. adding the last two blocks to go from ResNet-32 to ResNet-44), will lead to too many relevant residual blocks.
Again, this will make the transformed source ticket have a different structure than the target ticket and thus, will lead to worse performance.
Note that for the later two stages, the nature of the distribution of eigenvalues is different (Tablesand 8 of Appendix C).
In particular, stage 2 has only relevant and marginal blocks, and stage 3 has only marginal and non-relevant blocks.
However, there is consistent evidence for the fact that the number of each type of block is related to the three results from the original E-LTH paper (Chen et al., 2021c)
we highlighted.
D ISCUSSION
Inspired by similarities between the current state of sparse machine learning and the state of statistical physics in the early-to-mid–20 th century, we proved that iterative magnitude pruning (IMP), the principle method used to discover winning tickets, is a renormalization group (RG) scheme.
Given that the development of the RG led to a ﬁrst principled understanding of universal behavior near phase transitions, as well as a way in which to characterize materials by such behavior, we reasoned that viewing IMP from an RG perspective may lead to new insight on the universality of winning tickets (Desai et al., 2019; Mehta, 2019; Morcos et al., 2019; Soelen & Sheppard, 2019; Chen et al., 2020a; Gohil et al., 2020; Chen et al., 2021a; Sabatelli et al., 2021) and the general success IMP has found in the study of DNNs (Frankle et al., 2020b).
By viewing IMP as inducing a ﬂow in DNN parameter space (Fig. 2), we showed that ResNet50, pre-trained on ImageNet, has trajectories with similar properties for different tasks (Table 2).
As winning tickets are known to be transferable between these tasks Chen et al. (2021a), these results support a key prediction of the theory developed in Sec. 2.
Applying the RG framework to recent work which extended the notion of winning ticket universality by transforming tickets between different architectures (“Elastic Lottery Ticket Hypothesis”), we again found that the IMP eigenvalues allowed for a new, quantitative perspective on experimental results (Table 3).
While RG theory may be unfamiliar, and simpler explanations to the phenomena may conceivably exist, to our knowledge, no current theory has provided as general and successful a framework for understanding winning ticket universality, as what we have present here.
A wealth of theory has been developed around the RG, including a number of numerical and theoretical tools that go beyond what we have used.
Given the connection between IMP and the RG made explicit in this work, we believe that there is considerable potential for collaboration between the two ﬁelds.
Possible directions of future study include:
• Bringing the RG framework to other sparsiﬁciation methods, which may potentially similarly be viewed as RG schemes; • Bringing the RG framework to study winning tickets beyond computer vision models, such as in natural language processing (Chen et al., 2020a; Prasanna et al., 2020; Yu et al., 2020), reinforcement learning (Yu et al., 2020), and lifelong learning (Chen et al., 2021b); • Computing and classifying systems by their critical exponents (such as γ in Eq. 1).
We attempted to do this, but we were limited by having only a few independent seeds and a scaling function with multiple free parameters (Rosenfeld et al., 2020). This led to large differences between computed critical exponents of systems that displayed qualitatively similar scaling behavior (see Appendix D).
Using more advanced methods, such as ﬁnite scaling (Goldenfeld, 2005), and increasing the number of independent seeds may alleviate this problem; • Exploring the connection between the RG framework developed here and work done to build an effective theory of DNN behavior (Roberts et al., 2021).
Finally, while RG theory provides a way to know whether two given tasks, or architectures, can have winning tickets transferred between them, knowing what the minimal density a transferable winning ticket can have, remains an open question.
In order to address this, ﬁnite size effects (Goldenfeld, 2005), differences in symmetries, and non-linear corrections to the IMP ﬂow (Raju et al., 2019) may need to be taken into account.
This interesting, and important, question will be the focus of future work.
A A DDITIONAL R ES N ET -50 T RANSFER R ESULTS
In the main text we computed IMP eigenvalues for ResNet- pre-trained on ImageNet using simCLR self-supervised training (Chen et al., 2020b). These models were shown to have the best transferability of winning tickets across tasks in Chen et al. (2021a).
However, other, pre-trained models were also found to have good transferability. Here we present the same analysis for ResNet50, pre-trained on ImageNet using supervised training (Huh et al., 2016), again ﬁnding that the winning tickets for each task have similar distributions of eigenvalues (Table 4).
The data used for this was also taken from Chen et al. (2021a).
[Table]
B A DDITIONAL E VIDENCE OF IMP E IGENFUNCTIONS IN VGG-16 AND BERT
In the main text, we presented results for ResNets. However, there is no explicit assumptions of architecture present in the theory developed in Sec. 2. Therefore, we explored whether the functions
M i (Eq. 13) were similarly eigenfunctions for VGG-16, trained on CIFAR-10, and BERT, trained on the self-supervised mask language modeling (MLM) task.
Data is from experiments performed by Zhang et al. (2021) and Chen et al. (2020a), respectively. As can be seen in Tables 5 and 6, for both pairs of architectures and tasks, we ﬁnd that the computed eigenvalues have small (usually < 5% ) standard deviations, suggesting that it is reasonable to consider them as eigenfunctions of I .
Note that we only considered the ﬁrst 17 applications of IMP (matching that of the ResNet experiments presented in Table 2), in the case of VGG-16.
Considering additional applications led to higher standard deviations, which is expected as the size of the system decreases and ﬁnite size effects start to become non-negligible.
Interestingly, while VGG has layers that are relevant, marginal, and irrelevant, like ResNet-50,
BERT has only marginal layers.
This suggests that BERT may have been pushed into a ﬁxed point of the IMP ﬂow.
Whether, and how, this is related to the challenges associated with going deeper with Transformers, will be studied of future work.
[Table]
Coloring is same as Table 4.
[Table]
C A DDITIONAL E-LTH R ESULTS
In the main text on the, the experimental results surrounding the “Elastic Lottery Ticket Hypothesis” focused only on the eigenvalues corresponding to the eigenfunctions of residual blocks in the ﬁrst stage.
The models considered in Chen et al.
(2021c) had two other stages, whose eigenvalues we report here in Tables 7 and 8.
While the distribution of eigenvalues is different than it is in stage 1 (i.e. considerably fewer relevant residual blocks) [Table] they again support the idea that the distributions are related to the results from Chen et al. (2021c) that we focused on in the main text.
[Table] [Table]
Following the recent work on scaling laws for IMP developed by Rosenfeld et al.
[Table]
we ﬁt the error, (cid:15) , deﬁned as 100% minus the top-1 accuracy, as a function of density, d , deﬁned as percent of weights remaining, via the functional form
[Equation]
where i = √− 1 .
Here (cid:15) np is interpreted as the error associated with not pruning, (cid:15) ↑ as the asymptotic error upon maximal pruning, γ as the sensitivity of the combination of network architecture, task, activation function, and optimizer to pruning, and p as controlling how the transition from no change in error to power-law scaling takes place.
Importantly, γ can be viewed as a critical exponent under the RG framework.
Because systems in the same universality class have the same critical exponents, we wondered if we could ﬁnd evidence of universality in the computed value of γ for various different computer vision models.
We found the ﬁts to be sensitive to (cid:15)
np and, in some cases, there was not a clear
[Table]
The computed values of γ for ResNet-50 evaluated on CIFAR-10, CIFAR-100, and SVHN are presented in Table 9.
The effect of pre-training using various state-of-the-art methods is also given.
The γ values vary, sometimes considerably, across tasks and method of pre-training (e.g. CIFARand SVHN pre-trained via simCLR).
However, we note that the data came from only two or three seeds, making the ﬁts susceptible to noise and making the quantiﬁcation of error in the computed γ difﬁcult.
In addition, the ﬁtting function of Eq.
14 has multiple free parameters (including an additional one we added), again making the ﬁt susceptible to noise.
Taking these possible complications into account, we overlaid the different error as a function of density scaling curves (an example of which is given in Fig 3).
We found that they qualitatively matched each other more than the computed γ may have made it seem.
We imagine that trying different ﬁtting procedures (especially making use of methods developed in the context of statistical physics, such as ﬁnite scaling (Goldenfeld, 2005)), as well as using more independent random seeds, will enable more accurate and robust approximations of γ .