Beyond neural scaling laws: beating power law scaling via data pruning
Anonymous Author(s)
AffiliationAddress email
Abstract
Widely observed neural scaling laws, in which error falls off as a power of thetraining set size, model size, or both, have driven substantial performance improvements in deep learning.
However, these improvements through scaling alonerequire considerable costs in compute and energy.
Here we focus on the scalingof error with dataset size and show how both in theory and practice we can breakbeyond power law scaling and reduce it to exponential scaling instead if we haveaccess to a high-quality data pruning metric that ranks the order in which trainingexamples should be discarded to achieve any pruned dataset size.
We then test thisnew exponential scaling prediction with pruned dataset size empirically, and indeedobserve better than power law scaling performance on ResNets trained on CIFAR10, SVHN, and ImageNet.
Given the importance of finding high-quality pruningmetrics, we perform the first large-scale benchmarking study of ten different datapruning metrics on ImageNet.
We find most existing high performing metricsscale poorly to ImageNet, while the best are computationally intensive and requirelabels for every image.
We therefore developed a new simple, cheap and scalableself-supervised pruning metric that demonstrates comparable performance to thebest supervised metrics.
Overall, our work suggests that the discovery of gooddata-pruning metrics may provide a viable path forward to substantially improvedneural scaling laws, thereby reducing the resource costs of modern deep learning.
Introduction
Empirically observed neural scaling laws [1, 2, 3, 4, 5, 6, 7] in many domains of machine learning,including vision, language and speech, demonstrate that test error often falls off as a power law
witheither the amount of training data, model size or compute.
Such power law scaling has motivatedsignificant societal investments in data collection, compute, and associated energy consumption.
However, power law scaling is extremely weak and unsustainable.
For example, a drop in errorfrom 3% to 2% might require an order of magnitude more data, compute, or energy.
In languagemodeling with large transformers, a drop in cross entropy loss from about 3.4 to 2.8 nats requires 10times more training data (Fig. 1 in [2]).
Also, for large vision transformers, an additional 2 billionpre-training data points (starting from 1 billion) leads to an accuracy gain on ImageNet of a fewpercentage points (Fig.
1 in [6]).
Here we ask whether we might be able to do better.
For example,can we achieve exponential scaling instead, with a good strategy for selecting training examples?
Such vastly superior scaling would mean that we could go from 3% to 2% error by only adding a fewcarefully chosen training examples, rather than collecting 10 × more random ones.
Focusing on scaling of performance with training dataset size, we demonstrate that exponentialscaling is possible, both in theory and practice.
The key idea is that power law scaling of error
[Figure]
E
[Figure]
with respect to data suggests that many training examples are highly redundant.
Thus one shouldin principle be able to prune training datasets to much smaller sizes and train on the smaller pruneddatasets without sacrificing performance.
Indeed some recent works [8, 9, 10] have demonstratedthis possibility by suggesting various metrics to sort training examples in order of their difficulty orimportance, ranging from easy or redundant examples to hard or important ones, and pruning datasetsby retaining some fraction of the hardest examples.
However, these works leave open fundamentaltheoretical and empirical questions: When and why is successful data pruning possible?
What aregood metrics and strategies for data pruning?
Can such strategies beat power law scaling?
Can theyscale to ImageNet?
Can we leverage large unlabeled datasets to successfully prune labeled datasets?
We address these questions through both theory and experiment.
Our main contributions are:
Employing statistical mechanics, we develop a new analytic theory of data pruning in the
student-teacher setting for perceptron learning, where examples are pruned based on their
teacher margin, with large (small) margins corresponding to easy (hard) examples.
Our
theory quantitatively matches numerical experiments and reveals two striking predictions:
(a) The optimal pruning strategy changes depending on the amount of initial data; withabundant (scarce) initial data, one should retain only hard (easy) examples.
(b) Exponential scaling is possible with respect to pruned dataset size provided one chooses
an increasing Pareto optimal pruning fraction as a function of initial dataset size.
We show that the two striking predictions derived from theory hold also in practice in much
more general settings.
Indeed we empirically demonstrate signatures of exponential scaling
of error
with respect to pruned dataset size for ResNets trained from scratch on SVHN,
CIFAR-10 and ImageNet, and Vision Transformers fine-tuned on CIFAR-
Motivated by the importance of finding good quality metrics for data pruning, we perform alarge scale benchmarking study of 10 different data pruning metrics at scale on ImageNet,finding that most perform poorly, with the exception of the most compute intensive metrics.4.
We leveraged self-supervised learning (SSL) to developed a new, cheap unsupervised datapruning metric that does not require labels, unlike prior metrics.
We show this unsupervisedmetric performs comparably to the best supervised pruning metrics that require labels and much more compute.
This result opens the door to the exciting possibility of leveragingpre-trained foundation models to prune new datasets even before they are labeled.
Overall these results shed theoretical and empirical insights into the nature of data in deep learningand our ability to prune it, and suggest our current practice of collecting extremely large datasets maybe highly inefficient.
Our initial results in beating power law scaling motivate further studies andinvestments in not just inefficently collecting large amounts of random data, but rather, intelligentlycollecting much smaller amounts of carefully selected data, potentially leading to the creation anddissemination of foundation datasets , in addition to foundation models [11].
Background and related work
Our work brings together 3 largely disparate strands of intellectual inquiry in machine learning: (1)explorations of different metrics for quantifying differences between individual training examples;(2)
the empirical observation of neural scaling laws; and (3) the statistical mechanics of learning.
2.1 Pruning metrics: not all training examples are created equal
Several recent works have explored various metrics for quantifying individual differences betweendata points.
To describe these metrics in a uniform manner, we will think of all of them as orderingdata points by their difficulty, ranging from “easiest” to “hardest.”
When these metrics have beenused for data pruning, the hardest examples are retained, while the easiest ones are pruned away.
EL2N scores.
For example [9] trained small ensembles (of about 10 ) networks for a very shorttime (about 10 epochs) and computed for every training example the average L 2 norm of the errorvector (EL2N score).
Data pruning by retaining only the hardest examples with largest error enabledtraining from scratch on only 50% and 75% of CIFAR-10 and CIFAR-100 respectively without anyloss in final test accuracy.
However the performance of EL2N on ImageNet has not yet been explored.
Forgetting scores and classification margins.
[8] noticed that over the entire course of training,some examples are learned early and never forgotten, while others can be learned and unlearned (i.e.forgotten) repeatedly.
They developed a forgetting score which measures the degree of forgetting ofeach example.
Intuitively examples with low (high) forgetting scores can be thought of as easy (hard)examples.
[8] explored data pruning using these metrics, but not at ImageNet scale.
Memorization and influence.
[12] defined a memorization score for each example, correspondingto how much the probability of predicting the correct label for the example increases when it is presentin the training set relative to when it is absent; a large increase means the example must be memorized(i.e.
the remaining training data do not suffice to correctly learn this example).
Additionally [12] alsoconsidered an influence score that quantifies how much adding a particular example to the trainingset increases the probability of the correct class label of a test example.
Intuitively, low memorizationand influence scores correspond to easy examples that are redundant with the rest of the data, whilehigh scores correspond to hard examples that must be individually learned.
[12] did not use thesescores for data pruning as their computation is expensive.
We note since memorization explicitlyapproximates the increase in test loss due to removing each individual example, it is likely to be agood pruning metric (though it does not consider interactions).
Ensemble active learning.
Active learning iterates between training a model and selecting newinputs to be labeled [13, 14, 15, 16, 17].
In contrast, we focus on data pruning: one-shot selectionof a data subset sufficient to train to high accuracy from scratch.
A variety of coreset algorithms(e.g.
[18]) have been proposed for this, but their computation is expensive, and so data-pruning hasbeen less explored at scale on ImageNet.
An early clustering approach [19] allowed training on90% of ImageNet without sacrificing accuracy.
Notably [10] reduced this to 80% by training a largeensemble of networks on ImageNet and using ensemble uncertainty to define the difficulty of eachexample, with low (high) uncertainty corresponding to easy (hard) examples.
We will show how toachieve similar pruning performance without labels or the need to train a large ensemble.
Diverse ensembles (DDD).
[20] assigned a score to every ImageNet image, given by the number ofmodels in a diverse ensemble (10 models) that misclassified the image.
Intuitively, low (high) scorescorrespond to easy (hard) examples.
The pruning performance of this metric remains unexplored.
Summary.
We note: (1) only one of these metrics has tested well for its efficacy in data pruningat scale on ImageNet; (2) all of these metrics require label information; (3) there is no theory ofwhen and why data pruning is possible for any of these metrics; and (4) none of these works suggestthe possibility of exponential scaling.
We thus go beyond this prior work by benchmarking the datapruning efficacy of not only these metrics but also a new unsupervised metric we introduce thatdoes not require label information, all at scale on ImageNet.
We also develop an analytic theory fordata-pruning for the margin metric that predicts not only the possibility of exponential scaling butalso the novel finding that retaining easy instead of hard examples is better when data is scarce.
2.2 Neural scaling laws and their potential inefficiency
Recent work [1, 2, 3, 4, 5, 7] has demonstrated that test loss L often falls off as a power law withdifferent resources like model parameters ( N ), number of training examples ( P ), and amount ofcompute ( C ).
However, the exponents ν of these power laws are often close to 0 , suggestingpotentially inefficient use of resources.
For example, for large models with lots of compute, sothat the amount of training data constitutes a performance bottleneck, the loss scales as L ≈ P − ν .
Specifically for a large transformer based language model, ν = 0 .
095 , which implies an orderof magnitude increase in training data drops cross-entropy loss by only about 0 .
6 nats (Fig. 1 in[2]).
In neural machine translation experiments ν varies across language pairs from 0 .
35 to 0 .
48(Table 1 in [4]).
Interestingly, [7] explored a fixed computation budget C and optimized jointlyover model size N and training set size P , revealing that scaling both N and P commensurately as
C increases is compute optimal, and can yield smaller high performing models (trained on moredata) than previous work.
Nevertheless, for a transformer based language model, a 100 × increase incompute, corresponding to 10 × increases in both model size and training set size, leads to a drop incross-entropy loss of only about 0 .
5 nats (Fig. 2 in [7]).
Similar slow scaling holds for large visiontransformers where adding 2 billion pre-training images reduces ImageNet performance by a fewpercentage points (Fig. 1 in [6]).
While all of these results constitute significant improvements inperformance, they do come at a substantial resource cost whose fundamental origin arises from powerlaw scaling with small exponents.
Here, in the specific case of data scaling, we explore whether wecan beat power law scaling through careful data selection.
2.3 Statistical mechanics of perceptron learning
Statistical mechanics has long played a role in analyzing machine learning problems (see e.g.[21, 22, 23, 24] for reviews).
One of the most fundamental applications is perceptron learning inthe student-teacher setting [25, 26], in which random i.i.d.
Gaussian inputs are labeled by a teacherperceptron to construct a training set.
The test error for another student perceptron learning from thistraining set then scales as a power law with exponent − 1 for such data.
Such perceptrons have alsobeen analyzed in an active learning setting where the learner is free to design any new input to belabeled [27, 28], rather than choose from a fixed set of inputs, as in data-pruning.
Recent work [29]has analyzed this scenario but focused on message passing algorithms that are tailored to the case of
Gaussian inputs and perceptrons, and are hard to generalize to real world settings.
In contrast weanalyze margin based pruning algorithms that are used in practice in diverse settings, as in [8, 9].
An analytic theory of data pruning
To better understand data pruning, we employed the replica method from statistical mechanics [30] todevelop an analytic theory of pruning for the perceptron in the student-teacher setting [21] (see App.
Afor detailed derivations of all results).
Consider a training dataset of P examples { x µ , y µ } µ =1 ,...,Pwhere x µ ∈ R N are i.i.d.
zero mean unit variance random Gaussian inputs and y µ = sign ( T · x µ )are labels generated by a teacher perceptron with weight vector T ∈ R N .
We work in the highdimensional statistics limit where N, P → ∞ but the ratio α tot = PN of the number of total trainingexamples to parameters remains O (1) .
We then consider a pruning algorithm used in [8, 9], namely:(1) train a probe student perceptron for very few epochs on the training data, obtaining weights J probe ;(2) compute the margin m µ =
J probe · ( y µ x µ ) of each training example, where large (small) marginscorrespond to easy (hard) examples; (3) construct a pruned dataset of size P prune = fP , where f isthe fraction of examples kept, by retaining the P prune hardest examples, (4) train a new perceptron tocompletion on the smaller dataset with a smaller ratio α prune =
P prune N of examples to parameters.
[Figure]
We are interested in the test error ε of this final perceptron as a function of α tot , f , and the angleθ between the probe student J probe and the teacher T .
Our theory approximates J probe as simply arandom Gaussian vector conditioned to have angle θ with the teacher T .
Under this approximationwe obtain an analytic theory for ε ( α tot , f, θ ) that is asymptotically exact in the high dimensional limit(App.
A).
We first examine results when θ = 0 , so we are pruning training examples according totheir veridical margins with respect to the teacher (Fig.
1A).
We find two striking phenomena, eachof which constitute predictions in real-world settings that we will successfully confirm empirically.
The best pruning strategy depends on the amount of initial data.
First, we note the test errorcurve for f = 1 in Fig.
1A corresponding to no pruning, or equivalently to randomly pruning alarger dataset of size α tot down to a size α prune , exhibits the well known classical perceptron learningpower law scaling ε ∝ α − 1 prune .
Interestingly though, for small α tot , keeping the hardest examplesperforms worse than random pruning (lighter curves above darkest curve for small α prune in Fig.
1A).
However, for large α tot , keeping the hardest examples performs substantially better than randompruning (lighter curves below darkest curve for large α prune in Fig.
1A).
It turns out keeping the easiestrather than hardest examples is a better pruning strategy when α tot is small (Fig. 1C).
Intuitively, ifone does not have much data to start with, it is better to keep the easiest examples with largest margins(i.e.
the blue regions of Fig.
1B) to avoid overfitting.
Otherwise, if one starts with lots of data, sooverfitting is less of an issue, it is best to keep the hardest examples with smallest margin that providemore information about the teacher’s decision boundary (i.e. the green region of Fig.
1B).
Fig.
1Creveals which pruning strategy is best as a joint function of α tot and f .
Note the transition betweenoptimal strategies becomes sharper at small fractions f of data kept.
This transition between optimalpruning strategies can be viewed as a prediction in more general settings.
To test this prediction wetrained a ResNet18 on pruned subsets of the CIFAR-10 dataset (Fig.
1D), and observed strikinglysimilar behavior, indicating the prediction can hold far more generally, beyond perceptron learning.
Interestingly, [8, 9] missed this transition, likely because they started pruning from large datasets.
Pareto optimal data pruning can beat power law scaling.
A second prediction of our theoryis that when keeping a fixed fraction f of the hardest examples as α tot increases (i.e. constant colorcurves in Fig.
1A), the error initially drops exponentially in α prune = fα tot , but then settles intothe universal power law ε ∝ α − 1 prune for all fixed f .
Thus there is no asymptotic advantage to datapruning at a fixed f .
However, by pruning more aggressively (smaller f ) when given more initial data(larger α tot ), one can achieve a Pareto optimal test error as a function of pruned dataset size α prune thatremarkably traces out at least an exponential scaling law (Fig.
1A, purple curve).
Indeed our theorypredicts for each α prune a Pareto optimal point in α tot and f (subject to α prune = fα tot ), yielding forevery fixed α prune an optimal f opt , plotted in Fig.
1E. Note f opt decreases with α prune indicating moreaggressive pruning (smaller f opt ) of original datasets of larger size α tot is required to obtain larger
Pareto optimal pruned datasets of size α prune .
We will test this striking scaling prediction in Fig.
Beating power law scaling: an information-theoretic perspective.
Classical randomly selecteddata generates slow power law error scaling because each extra training example provides less newinformation about the correct decision boundary than the previous example.
More formally, let S ( α tot )denote the typical entropy of the posterior distribution over student perceptron weights consistentwith a training set of size α tot .
The information gain I ( α tot ) due to additional examples beyond α tot
[Figure]
can be defined as the rate at which the posterior entropy is reduced:
I ( α tot ) = − d dα tot S ( α tot ) .
Inclassical perceptron learning I ( α tot ) decays to zero as a power law in α tot , reflecting a vanishingamount of information per each new example, leading to the slow power law decay of test errorε ∝ α − 1 tot .
However, data pruning can increase the information gained per example by pruning awaythe uninformative examples.
To show this, we generalized the replica calculation of the posteriorentropy S and information gain I from random datasets of size α tot to pruned datasets of size α prune(App.
A).
We plot the resulting information gain I ( α prune ) for different f in Fig.
1F. For any fixedf , I ( α prune ) will eventually decay as a power law as α − 1 prune .
However, by more aggressively pruning(smaller f ) datasets of larger size α tot , I ( α prune ) can converge to a finite value I ( ∞ )
= 1 nat/example,resulting in larger pruned datasets only adding useful non-redundant information.
Since each newexample under Pareto optimal data pruning conveys finite information about the target decisionboundary, as seen in Fig.
1F, the test error can decay at least exponentially in pruned dataset size asin Fig.
1A.
An imperfect pruning metric yields a cross over from exponential to power law scaling.
Wenext examine the case of nonzero angle θ between the probe student J probe and the teacher T , suchthat the ranking of training examples by margin is no longer completely accurate (Fig. 2A).
Retainingthe hard examples with smallest margin with respect to the probe student will always result in pruneddatasets lying near the probe’s decision boundary.
But if θ is large, such examples might be far fromthe teacher’s decision boundary, and therefore could be less informative about the teacher (Fig.
2A).
As a result our theory, confirmed by simulations, predicts that under nonzero angles θ , the Paretooptimal lower envelope of test error over both α tot and f initially scales exponentially as a functionof α prune = fα tot
but then crosses over to a power law (Fig. 2BCD).
Indeed, at any given nonzero θ ,our theory reveals that as α tot (and therefore α prune ) becomes large, one cannot decrease test error anyfurther by retaining less than a minimum fraction f min ( θ ) of all available data.
For example whenθ = 10 ◦ ( θ = 20 ◦ ) one can do no better asymptotically than pruning down to 24% ( 46% ) of the totaldata (Fig.
2CD).
As θ approaches 0 , f min ( θ ) approaches 0 , indicating that one can prune extremelyaggressively to arbitrarily small f while still improving performance, leading to at least exponentialscaling for arbitrarily large α prune in Fig.
2B.
However, for nonzero θ , the lack of improvementfor f < f min ( θ ) at large α prune renders aggressive pruning ineffective.
This result highlights theimportance of finding high quality pruning metrics with θ ≈ 0 .
Such metrics can delay the cross overfrom exponential to power law scaling as pruned dataset size α prune increases, by making aggressivepruning with very small f highly effective.
Strikingly, in App.
Fig.
10 we demonstrate this cross-overin a real-world setting by showing that the test error on SVHN is bounded below by a power lawwhen the dataset is pruned by a probe ResNet18 under the EL2N metric, trained for 4 epochs (weakpruning metric) but not a probe ResNet18 trained for 40 epochs (strong pruning metric).
Data pruning can beat power law scaling in practice
Our theory of data pruning for the perceptron makes three striking predictions which can be tested inmore general settings, such as deep neural networks trained on benchmark datasets: (1) relative torandom data pruning, keeping only the hardest examples should help when the initial dataset size
T e s t
e rr o r o n C
I F
A RFine-tuning examples
Paretofrontier
A
B
Frac. of ImageNet used for pretraining
ResNet50 finetuned on
CIFART e s t a cc u r a c y ( % ) 0.75 1. is large, but hurt when it is small; (2) data pruning by retaining a fixed fraction f of the hardestexamples should yield power law scaling, with exponent equal to that of random pruning, as theinitial dataset size increases; (3) the test error optimized over both initial data set size and fractionof data kept can trace out a Pareto optimal lower envelope that beats power law scaling of test erroras a function of pruned dataset size, through more aggressive pruning at larger initial dataset size.
We verified all three of these predictions on ResNets trained on SVHN, CIFAR-10, and ImageNetusing varying amounts of initial dataset size and fractions of data kept under data pruning (comparetheory in Fig.
3A with deep learning experiments in Fig.
3BCD).
In each experimental setting we seebetter than power law scaling at larger initial data set sizes and more aggressive pruning.
Moreoverwe would likely see even better scaling with even larger initial datasets (as in Fig.3A dashed lines).
Data pruning improves transfer learning.
Modern foundation models are pre-trained on a largeinitial dataset, and then transferred to other downstream tasks by fine-tuning on them.
We thereforeexamined whether data-pruning can be effective for both reducing the amount of fine-tuning data andthe amount of pre-training data.
To this end, we first analyzed a vision transformer (ViT) pre-trained on
ImageNet21K and then fine-tuned on different pruned subsets of CIFAR-10.
Interestingly, pre-trainedmodels allow for far more aggressive data pruning; fine-tuning on only 10% of CIFAR-10 can matchor exceed performance obtained by fine tuning on all of CIFAR-10 (Fig. 4A).
Furthermore Fig.
4Aprovides a new example of beating power law scaling in the setting of fine-tuning.
Additionally, weexamined the efficacy of pruning pre-training data by pre-training ResNet50s on different prunedsubsets of ImageNet1K (exactly as in Fig.
3D) and then fine-tuning them on all of CIFAR-10.
Fig.
4B demonstrates pre-training on as little as 50% of ImageNet can match or exceed CIFAR-10performance obtained by pre-training on all of ImageNet.
Thus intriguingly pruning pre-training dataon an upstream task can still maintain high performance on a different downstream task.
Overallthese results demonstrate the promise of data pruning in transfer learning for both the pre-trainingand fine-tuning phases.
Benchmarking supervised pruning metrics on ImageNet
We note that the majority of data pruning experiments have been performed on small-scale datasets(i.e.
variants of MNIST and CIFAR), while the few pruning metrics proposed for ImageNet haverarely been compared against baselines designed on smaller datasets.
Therefore, it is currently unclearhow most pruning methods scale to ImageNet and which method is best.
Motivated by how stronglythe quality of a pruning metric can impact performance in theory (Fig. 2), we decided to fill thisknowledge gap by performing a systematic evaluation of 8 different supervised pruning metrics on
ImageNet: two variants of influence scores [12], two variants of EL2N [9], DDD [20], memorization[12], ensemble active learning [10], and forgetting [8].
See Section 2 for a review of these metrics.
Additionally, we include two new prototypicality metrics that we introduce in the next section.
We first asked how consistent the rankings induced by different metrics are by computing the
Spearman rank correlation between each pair of metrics (Fig.
5A).
Interestingly, we found substantialdiversity across metrics, though some (EL2N, DDD, and memorization) were fairly similar with rankcorrelations above 0 .
However, we observed marked performance differences between metrics:
Fig 5BC shows test performance when a fraction f of the hardest examples under each metric arekept in the training set.
Despite the success of many of these metrics on smaller datasets, only a fewstill match performance obtained by training on the full dataset, when selecting a significantly smallertraining subset (i.e. about 80% of ImageNet).
Nonetheless, most metrics continue to beat randompruning, with memorization in particular demonstrating strong performance (Fig. 5C).
A
C
B
A
C
B
A
C
B
A
C
B
A
C
B
A
C
B A B
C
T o p - 5 a cc u r a c y ( % ) T o p - 5 a cc u r a c y ( % )
Fraction of data kept
Fraction of data kept s e l f - s u p e
r
v i s e d p r o t o t y p
e s i n fl u e n
c e s u m - a b s s u p e
r
v i s e d p r o t o t y p
e s
We found that all pruning metrics amplify class imbalance, which results in degraded performance.
To solve this we used a simple 50% class balancing ratio for all ImageNet experiments.
Furtherdetails and baselines without class balancing are shown in App. H.
Self-supervised data pruning through a prototypicality metric
Fig.
5 shows many data pruning metrics do not scale well to ImageNet, while the few that do requiresubstantial amounts of compute.
Furthermore, all these metrics require labels, thereby limiting theirability to prune data for large-scale foundation models trained on massive unlabeled datasets [11].
Thus there is a clear need for simple, scalable, self-supervised pruning metrics.
To compute a self-supervised pruning metric for ImageNet, we perform k -means clustering in theembedding space of an ImageNet pre-trained self-supervised model (here: SWaV [31]), and definethe difficulty of each data point by the distance to its nearest cluster centroid, or prototype.
Thuseasy (hard) examples are the most (least) prototypical.
Encouragingly, in Fig.
5C, we find ourself-supervised prototype metric matches or exceeds the performance of the best supervised metric,memorization, until only 70% of the data is kept, despite the fact that our metric does not use labelsand is much simpler and cheaper to compute than many previously proposed supervised metrics.
See
App. Fig.
9 for further scaling experiments using the self-supervised metric.
To assess whether the clusters found by our metric align with ImageNet classes, we compared theiroverlaps in Fig.
6A.
Interestingly, we found alignment for some but not all classes.
For example, classcategories such as snakes were largely aligned to a small number of unsupervised clusters, whileother classes were dispersed across many such clusters.
If class information is available, we canenforce alignment between clusters and classes by simply computing a single prototype for eachclass (by averaging the embeddings of all examples of this class).
While originally intended to be anadditional baseline metric (called supervised prototypes, light blue in Fig 5BC), this metric remarkablyoutperforms all other supervised metrics and largely matches the performance of memorization, whichis prohibitively expensive to compute.
Moreover, when more than 70% of the data are kept, theperformance of the self-supervised and supervised metrics are similar, demonstrating the promise ofself-supervised pruning.
One important choice for the self-supervised prototype metric is the number of clusters k .
We found,reassuringly, our results were robust to this choice: k can deviate one order of magnitude more orless than the true number of classes (i.e. 1000 for ImageNet) without affecting performance (App. F).
To better understand example difficulty under various metrics, we visualize extremal images for ourself-supervised prototype metric and the memorization metric for one class (Fig 6B,C).
Qualitatively,easy examples correspond to highly similar, redundant images, while hard examples look likeidiosyncratic outliers.
See App.
E, Figs.
11,12,13,14,15,16,17,18 for more classes and metrics.
[Figure]
L c l u s t e r ( s o r t e d h
i e r a r c h
i c a ll y )
[Figure]
Discussion
[Figure]
Summary.
We have shown, both in theory and practice, how to break beyond slow power lawscaling of error versus dataset size to faster exponential scaling, through data pruning.
Additionally wehave developed a simple self-supervised pruning metric that enables us to discard 20% of ImageNetwithout sacrificing performance, on par with the best and most compute intensive supervised metric.
Limitations.
The most notable limitation is that achieving exponential scaling requires a highquality data pruning metric.
Since most metrics developed for smaller datasets scale poorly to
ImageNet, our results emphasize the importance of future work in identifying high quality, scalablemetrics.
Our self-supervised metric provides a strong initial baseline.
Moreover, a key advantage ofdata pruning is reduced computational cost due to training on a smaller dataset for the same numberof epochs as the full dataset (provided the computational cost of computing the pruning metric doesnot dominate).
However, we found that performance often increased when the same number trainingiterations on the full dataset was carried out on the pruned dataset, resulting in the same training time,but additional training epochs.
However, this performance gain saturated before training time on thepruned dataset approached that on the whole dataset (App. J) thereby still yielding a computationalefficiency gain.
Overall this tradeoff between accuracy and training time on pruned data is importantto consider in evaluating potential gains due to data pruning.
Finally, we found that class-balancingwas essential to maintain performance on data subsets (App. H).
Future work will be required toidentify ways to effectively select the appropriate amount of class-balancing.
Ethical considerations.
A potential negative societal impact could be that data-pruning leads tounfair outcomes for certain groups.
We have done a preliminary analysis of how data-pruning affectsperformance on individual ImageNet classes (App. I), finding no substantial differential effects acrossclasses.
However proper fairness tests specific to deployment settings should always be conducted onevery model, whether trained on pruned data or not.
Additionally, we analyzed the impact of pruningon OOD performance (App. K).
Outlook: Towards foundation datasets.
We believe the most promising future direction is thefurther development of scalable, unsupervised data pruning metrics.
Indeed our theory predicts thatthe application of pruning metrics on larger scale datasets should yield larger gains by allowing moreaggressive pruning.
This makes data pruning especially exciting for use on the massive unlabeleddatasets used to train large foundation models (e.g. 400M image-text pairs for CLIP [32], 3.5B
Instagram images [33], 650M images for the DALLE-2 encoder [34], 780B tokens for PALM[35]).
If highly pruned versions of these datasets can be used to train a large number of differentmodels, one can conceive of such carefully chosen data subsets as foundation datasets in whichthe initial computational cost of data pruning can be amortized across efficiency gains in trainingmany downstream models, just at the initial computational cost of training foundation models isamortized across the efficiency gains of fine-tuning across many downstream tasks.
Together, our
Checklist
For all authors...
[Table]
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
[Table]
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See discussion section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seediscussion section 7 and App.
I.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
[Table]
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See App.
A
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] See App.
A
[Table]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] All code for
theory plots and numerical perceptron simulations is packaged in a reproducible colab
[Table]
notebook.
[Table]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See App.
B.
[Table]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] Since we train a very large (order 100) number of models
on computationally intensive tasks (e
g. ImageNet), many of our plots contain only
results from a single random seed.
However, variability due to random seeds can be
inferred by the smooth progression of the relevant quantity on each plot.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See App.
B
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See App.
B
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See App.
B
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See App.
B
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
A.1 Problem setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Main result and overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Replica calculation of the generalization error . . . . . . . . . . . . . . . . . .
A.4 Quenched entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Perfect teacher-probe overlap . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6 Information gain per example . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.7 Imperfect teacher-probe overlap . . . . . . . . . . . . . . . . . . . . . . . . . .
A.8 Optimal pruning policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.9 Exact saddle point equations . .
B
[Table]
training method details & dataset information
C Breaking compute scaling laws via data pruning
D Additional scaling experiments
E Extremal images according to different metrics
F Impact of number of clusters k on self-supervised prototypes
G
Impact
of ensemble prototypes
H Relationship between pruning and class (im-)balance
I Effect of pruning on class-conditional accuracy and
[Table] [Table]
A A theory of data-pruning for the perceptron: detailed derivations
All code required to reproduce the theory figures and numerical simulations throughout this paper canbe run in the Colab notebook at https://colab.research.google.com/drive/1in35C6jh7y_ynwuWLBmGOWAgmUgpl8dF?usp=sharing .
A.1 Problem setup
In this section we introduce a theory for data pruning in the teacher-student perceptron setting, usingthe tools of statistical mechanics.
We study the problem of classifying a dataset of P examples{ x µ , y µ } µ =1 ,...,P , where x µ ∼ N (0 , I N ) are i.i.d.
zero mean unit variance random Gaussian inputs,and y µ = sign ( T · x ) are labels generated by a teacher perceptron T ∈ R N , which we will assumeis randomly drawn from a uniform distribution on the sphere T ∼ Unif ( S N − 1 ( √ N )) .
We work inthe high-dimensional statistics limit where N, P → ∞ but the ratio α tot = P/N remains O (1) .
Thegeneralization error of a perceptron trained on such an isotropic dataset is a classical problem (seee.g.
However, we are interested in the setting where the training dataset is not isotropic, butinstead has inherited some structure due to data pruning.
In particular, consider pruning the training dataset by keeping only the examples with the smallestmargin | z µ | = | J probe · x µ | along a probe student J probe .
The pruned dataset will follow somedistribution p ( z ) along the direction of J probe , and remain isotropic in the nullspace of J probe .
Inwhat follows we will derive a general theory for an arbitrary data distribution p ( z ) , and specializeto the case of small-margin pruning only at the very end (in which case p ( z ) will take the form of atruncated Gaussian).
We will also make no assumptions on the form of the probe student J probe or thelearning rule used to train it; only that J probe has developed some overlap with the teacher, quantifiedby the angle θ = cos − 1 (cid:0)
[Equation]
(Fig. 2 A ).
After the dataset has been pruned, we consider training a new student J from scratch on the pruneddataset.
A typical training algorithm (used in support vector machines and the solution to which
SGD converges on separable data) is to find the solution J which classifies the training data withthe maximal margin κ = min µ J · ( y µ x µ ) .
Our goal is to compute the generalization error ε gof this student, which is simply governed by the overlap between the student and the teacher,ε g = cos − 1 ( R )
/π , where R = J · T / ∥ J ∥ 2 ∥ T ∥ 2 .
A.2 Main result and overview
Our main result is a set of self-consistent equations which can be solved to obtain the generalizationerror ε ( α, p, θ ) for any α and any data distribution p ( z ) along a probe student at any angle θ relativeto the teacher.
These equations take the form,
[Equation]
Where,
[Equation]
Where ⟨·⟩ z represents an average over the pruned data distribution p ( z ) along the probe student.
For any α, p ( z ) , θ , these equations can be solved for the order parameters R, ρ, κ , from which thegeneralization error can be easily read off as ε g = cos − 1 ( R ) /π .
This calculation results in the solidtheory curves in Figs 1,2,3, which show an excellent match to numerical simulations.
In the followingsection we will walk through the derivation of these equations using replica theory.
In Section A.6we will derive an expression for the information gained per training example, and show that with
Pareto optimal data pruning this information gain can be made to converge to a finite rate, resultingin at least exponential decay in test error.
In Section A.7, we will show that super-exponential scalingeventually breaks down when the probe student does not match the teacher perfectly, resulting inpower law scaling at at a minimum pruning fraction f min ( θ ) .
A.3 Replica calculation of the generalization error
To obtain Eqs.
1,2,3, we follow the approach of Elizabeth Gardner and compute the volumeΩ( x µ , T , κ ) of solutions J which perfectly classify the training data up to a margin κ (known as the
Gardner volume) [25, 21].
As κ grows, the volume of solutions shrinks until it reaches a uniquesolution at a critical κ , the max-margin solution.
The Gardner volume Ω takes the form,
[Equation]
Because the student’s decision boundary is invariant to an overall scaling of J , we enforce normalization of J via the measure dµ ( J ) ,
[Equation]
In the thermodynamic limit N, P → ∞ the typical value of the entropy S ( κ ) =
⟨⟨ log Ω( x µ , T , κ ) ⟩⟩is dominated by particular values of R, κ , where the double angle brackets ⟨⟨·⟩⟩ denote a quenchedaverage over disorder introduced by random realizations of the training examples x µ and the teacher
T .
However, computing this quenched average is intractable since the integral over J cannot beperformed analytically for every individual realization of the examples.
We rely on the replica trickfrom statistical physics,
[Equation]
Which allows us to evaluate S ( κ ) in terms of easier-to-compute powers of Ω ,
[Equation]
This reduces our problem to computing powers of Ω , which for integer n can be written in terms ofα = 1 , . . .
, n replicated copies of the original system,
[Equation]
We begin by introducing auxiliary variables,
[Equation] [Equation]
by δ − functions, to pull the dependence on J and T outside of the Heaviside function,
[Equation]
Using the integral representation of the δ -functions,
[Equation]
[Equation]
The data obeys some distribution p ( z ) along the direction of J probe and is isotropic in the nullspace of
J probe .
Hence we can decompose a training example x µ as follows, x µ =
J probe z µ +( I − J probe J T probe ) s µ ,where z µ ∼ p ( z ) and s µ ∼ N (0 , I N ) ,
[Equation]
Where J ⊥ = (1 − J probe J T probe ) J and T ⊥
=
(1 − J probe J T probe ) T .
Now we can average over the patternss µ ∼ N (0 , I N ) ,
[Equation]
exp
[Equation] [Equation]
Inserting this back into our expression for the Gardner volume,
[Equation]
As is typical in replica calculations of this type
[Equation]
we now introduce order parameters,
[Equation]
which will allow us to decouple the J - from the λ - µ - z - integrals.
q αβ represents the overlaps betweenreplicated students, and R α the overlap between each replicated student and the teacher.
However,because our problem involves the additional role of the probe student, we must introduce an additionalorder parameter,
[Equation]
which represents the overlap between each replicated student and the probe student.
Notice that,
[Equation] [Equation]
With this new set of order parameters in hand
[Table]
we can decouple the J from the λ − u − z − integrals.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We can now perform the gaussian integral over ˆ u µ ,
[Equation]
Expanding,
[Table]
Simplifying,
[Table]
[Equation]
Now we introduce integral representations for the remaining delta functions
[Equation]
including the measuredµ ( J α )
[Equation]
for which we introduce the parameter ˆ k α ,
[Table]
[Equation]
Notice that the u µ − λ αµ − ˆ λ αµ − z µ -integrals factorize in µ
[Equation]
and can be written as a single integral tothe power of P = αN .
[Equation] [Equation]
Where we have written the Gardner volume in terms of an entropic part G S
[Equation]
which measures howmany spherical couplings satisfy the constraints,
[Equation]
And an energetic part G E ,
[Equation]
We first evaluate the entropic part, G S , by introducing the n × n matrices A, B ,
[Equation]
Inserting this our expression for G S becomes
[Equation]
Integrating over J α ,
[Equation]
Now we can include the remaining terms in the expression for Ω ( n ) outside of G E and G S by notingthat
[Equation]
Additionally, we can use log det A = tr (log A ) .
Thus all terms in the exponent except G E can bewritten as
[Equation]
Now we extremize wrt ˆ R α and the elements of A by setting the derivatives wrt ˆ R γ , ˆ ρ γ and A γδ equalto zero:
[Equation]
Solving these gives
[Equation]
and
[Equation]
and now we are left with
[Equation]
A.3.1 Replica symmetry ansatz
In order to extremize wrt q αβ , R α , ρ α , we take the replica symmetry ansatz [21],
[Equation]
Then C takes the form
[Equation] [Equation]
A matrix with E on the diagonal and F elsewhere, C αβ =
Eδ αβ + F (1 − δ αβ ) , has n − 1 degenerateeigenvalues E − F and one eigenvalue E + ( n − 1) F .
Hence in our case C has n − 1 degenerateeigenvalues
[Equation]
and one other eigenvalue,
[Equation]
Therefore,
[Equation]
We next evaluate the energetic part, G E ,
[Equation]
First note that we can rewrite the terms
[Equation] [Equation]
To simplify the last term we apply the Hubbard-Stratonovich transformation, e b 2 / 2 = (cid:82) Dte bt ,introducing auxiliary field t ,
[Equation]
Using the Θ -function to restrict the bounds of integration
[Equation] [Equation]
Now we can perform the gaussian integrals over ˆ λ α ,
[Equation]
z
[Equation]
And λ α ,= log
[Equation]
∞
[Equation] [Equation]
Shifting the integration variable t → ( ρ α z + ( u − z cos θ )( R α − ρ α cos θ ) sin 2 θ
[Equation]
√ q ,we can finally perform the gaussian integral over u ,
[Equation]
(cid:19)(cid:29)
[Equation]
We can simplify this further by taking t → ( √ qt − zρ ) / (cid:112) q − ρ 2 ,
[Equation]
log
[Equation]
A.4 Quenched entropy
Putting everything together, and using the replica identity, ⟨ log Ω ⟩ = lim n → 0 ( ⟨ Ω n ⟩ − 1)
/n, we obtainan expression for the quenched entropy of the teacher-student perceptron under data pruning:
[Equation]
We will now unpack this equation and use it to make predictions in several specific settings.
A.5 Perfect teacher-probe overlap
We will begin by considering the case where the probe student has learned to perfectly match theteacher, J probe =
T , which we can obtain by the limit θ → 0 , ρ → R .
In this limit the second
H -function in Eq.
63 becomes increasingly sharp, approaching a step function:
[Equation]
→ Θ( z ) (64)
Hence we are left with,
[Equation]
A.5.1 Saddle point equations
We can now obtain a set of self-consistent saddle point equations by setting set to zero the derivativeswith respect to R and q of the right side of Eq.
As κ approaches its critical value, the space ofsolutions shrinks to a unique solution, and hence the overlap between students
q approaches one.
Inthe limit q → 1 , after some partial integration, we find,
[Equation]
These saddle point equations can be solved numerically to find R and κ as a function of α for astudent perceptron trained on a dataset with an arbitrary distribution along the teacher direction p ( z ) .
We can specialize to the case of data pruning by setting p ( z ) to the distribution found after pruningan initially Gaussian-distributed dataset so that only a fraction f of those examples with the smallestmargin along the teacher are kept, p ( z ) =
e − z 2 / 2 √ 2 πf Θ( γ − | z | ) , where the threshold γ = H − 1 (cid:0) 1 − f 2 (cid:1) .
[Equation]
Solving these saddle point equations numerically for R and κ yields an excellent fit to numericalsimulations, as can be seen in Fig.
1 A .
It is also easy to verify that in the limit of no data pruning( f → 1 , γ → ∞ ) we recover the saddle point equations for the classical teacher-student perceptron(Eqs.
4.4 and 4.5 in [21]),
[Equation]
A.6 Information gain per example
Why does data pruning allow for super-exponential performance with dataset size α ?
We candefine the amount of information gained from each new example, I ( α ) , as the fraction by which thespace of solutions which perfectly classify the data is reduced when a new training example is added,
I ( α ) =
Ω( P +1 N ) / Ω( PN ) .
Or, equivalently, the rate at which the entropy is reduced, I ( α )
=
− ddα S ( α ) .
Of coure, the volume of solutions shrinks to zero at the max-margin solution; so to study the volumeof solutions which perfectly classify the data we simply set the margin to zero κ = 0 .
In [27] theinformation gain for a perceptron in the classical teacher-student setting is shown to take the form,
[Equation]
Which goes to zero in the limit of large α as I ( α ) ∼ 1 /α .
Data pruning can increase the informationgained per example by pruning away the uninformative examples.
To show this, we generalizethe calculation of the information gain to pruned datasets, using the expression for the entropy weobtained in the previous section (Eq. 65).
[Equation]
Hence the information gain I ( α ) =
− ddα S ( α ) is given by
[Equation]
Changing variables to t → − ( √ Rt +
[Equation] [Equation]
[Equation]
Now, assuming that we prune to a fraction f , so that p ( z ) =
Θ (cid:0)
| z | − γ (cid:1) exp( − z 2 / 2) √ 2 πf , where γ =
[Equation]
I ( α ) is plotted for varying values of f in Fig.
1 F .
Notice that for f → 1 , γ → ∞ and we recover Eq.72.
For f → 0 , γ → 0 and we obtain,
[Equation]
[Equation]
Which allows us to produce to trace the Pareto frontier in Fig.
1 F .
For R → 1 , Eq.
77 gives theasymptotic information gain I ( ∞ )
= 1 nat/example.
A.7 Imperfect teacher-probe overlap
In realistic settings we expect the probe student to have only partial information about the targetfunction.
What happens if the probe student does not perfectly match the teacher?
To understand thiscarefully, we need to compute the full set of saddle point equations over R, q, and ρ , which we willdo in the following section.
But to first get an idea for what goes wrong, we include in this section asimple sketch which reveals the limiting behavior.
Consider the case where the angle between the probe student and teacher is θ .
Rotate coordinates sothat the first canonical basis vector aligns with the student J = (1 , 0 , . . . , 0) , and the teacher lies inthe span of the first two canonical basis vectors, T
= (cos θ, sin θ, 0 , . .
Consider the marginalong the teacher of a new training example x drawn from the pruned distribution.
[Equation]
As the fraction of examples kept goes to zero, E x 20 → 0 , and the average margin of a new exampleconverges to a fixed value,
[Equation]
Hence the data ultimately stops concentrating around the teacher’s decision boundary, and theinformation gained from each new example goes to zero.
Therefore we expect the generalizationerror to converge to a power law, where the constant prefactor is roughly that of pruning with a prunefraction f min which yields an average margin of 1 − R 2 .
This “minimum” pruning fraction lowerbounds the generalization error envelope (see Fig. 2), and satisfies the following equation,
[Equation] [Equation]
where γ min = H − 1 (cid:18)
1 − f min(cid:19) .
Eq.
80 can be solved numerically, and we use it to produce thelower-bounding power laws shown in red in Fig.
2 C,D .
The minimum achievable pruning fractionf min ( θ ) approaches zero as the angle between the probe student and the teacher shrinks, and we canobtain its scaling by taking R → 1 , in which case we find, [Equation]
[Equation]
A.8 Optimal pruning policy
The saddle point equations Eq.
66,67 reveal that the optimal strategy for α prune large is to retain onlythe "hardest" (smallest-margin) examples.
A.9 Exact saddle point equations
To obtain exact expressions for the generalization error for all θ , we can extremize Eq.
63 wrt R, q, ρ .
[Figure]
Derivative wrt R
[Equation]
Where we have defined,
[Equation]
[Equation]
Integrating the right-hand side by parts,
[Equation]
[Equation]
where
[Equation] [Equation]
Changing variables to t → t (cid:112) q − ρ 2 + ρz and taking the limit q → 1 ,
[Equation]
Where with this change of variables,
[Equation]
Derivative wrt q ,
[Equation]
[Equation]
Where Γ( t, z ) = ( R − ρ cos θ ) (cid:16) t (cid:112) q − ρ 2 + ρz (cid:17) + qz cos θ − ρRz .
After integating by parts,
[Equation] [Equation]
Changing variables to t → t (cid:112) q − ρ 2 + ρz and taking the limit q → 1 ,
[Equation]
Where now Γ( t, z ) =
z ( ρR − cos θ ) − t
( R − ρ cos θ ) .
Derivative wrt ρ ,
[Equation] [Equation]
Integrating the second term by parts,
[Equation]
Changing variables to t → t (cid:112) q − ρ 2 + ρz and taking the limit q → 1 ,
[Equation]
So together we have three saddle point equations:
[Equation]
Where
[Equation] [Equation]
Solving these equations numerically yields an excellent fit to numerical simulations on structureddata (Fig. 2 BCD ).
B Model training method details & dataset information
Perceptron in the teacher-student setting All code to reproduce these simulations can be found at:https://colab.research.google.com/drive/1in35C6jh7y_ynwuWLBmGOWAgmUgpl8dF?
usp
=sharing .
Perceptrons were trained on a synthetic dataset of P examples { x µ , y µ } µ =1 ,...,P ,where x µ ∼ N (0 , I N ) are i.i.d.
zero mean unit variance random Gaussian inputs, and y µ = sign ( T · x )are labels generated by a teacher perceptron T ∈ R N , which was randomly drawn from a uniformdistribution on the sphere T ∼ Unif ( S N − 1 ( √ N )) .
For all of our experiments we fixed N = 200 andset P = αN where α varied between 10 0 . 1 and 10 0 . 5 .
Each synthetic dataset was pruned to keep afraction f of the smallest-margin examples, where f varied between 0 . 1 and 1 in Fig.
1 and between0 .
2 and 1 in Figs.
2,3 to match the real-world experiments in Fig.
Perceptrons were optimized tofind the max-margin separating solution using a standard quadratic programming (QP) algorithmfrom the CVXPY library (for analysis of the computational complexity of this algorithm see Fig.
Results were averaged over 100 independent draws of the teacher and training examples.
ImageNet.
ImageNet model training was performed using a standard ResNet-50 through the VISSLlibrary [36] (stable version v0.1.6 ), which provides default configuration files for supervised ResNet50 training (accessible here; released under the MIT license).
Each model was trained on a singlenode of 8 NVIDIA V100 32GB graphics cards with BATCHSIZE_PER_REPLICA = 256, using the
Stochastic Gradient Descent (SGD) optimizer with a base learning rate = 0.1, nesterov momentum= 0.9, and weight decay = 0.001.
For our scaling experiments (Fig. 3 and Fig. 9), we trained onemodel per fraction of data kept (0.1-1.0) for each dataset size.
In total, these plot required training 97models on (potentially a subset of) ImageNet.
All the models were trained with matched number ofiterations, corresponding to 105 epochs on the full ImageNet dataset.
The learning rate was decayedby a factor of 10 after the number of iterations corresponding to 30, 60, 90, and 100 epochs on thefull ImageNet dataset.
For our main ImageNet experiments (Fig. 5) we trained one model per fraction of data kept (1.0,0.9, 0.8, 0.7, 0.6) × metric (11 metrics in total).
In the plot itself, since any variation in the“fraction of data kept = 1.0” setting is due to random variation across runs not due to potentialmetric differences, we averaged model performances to obtain a single datapoint here (while alsokeeping track of the variation across models, which is plotted as ± 2 standard deviations).
In total,this plot required training 55 models on (potentially a subset of) ImageNet.
ImageNet [37] isreleased under the ImageNet terms of access.
It is important to note that ImageNet images areoften biased [38, 39].
The SWaV model used to compute our prototypicality metrics was obtainedvia torch.hub.load(‘facebookresearch/swav:main’, ‘resnet50’) , which is the originalmodel provided by [31]; we then used the avgpool layer’s activations.
CIFAR-10 and SVHN.
CIFAR-10 and SVHN model training was performed using a standard
ResNet-18 through the PyTorch library.
Each model was trained on a single NVIDIA TITAN Xp12GB graphics card with batch size = 128, using the Stochastic Gradient Descent (SGD) optimizerwith learning rate = 0.1, nesterov momentum = 0.9, and weight decay = 0.0005.
Probe models weretrained for 20 epochs each for CIFAR-10 and 40 epochs each for SVHN.
Pruning scores were thencomputed using the EL2Ns metric [9], averaged across 10 independent initializations of the probemodels.
To evaluate data pruning performance, fresh models were trained from scratch on eachpruned dataset for 200 epochs, with the learning rate decayed by a factor of 5 after 60, 120 and 160epochs.
Data pruning for transfer learning.
To assess the effect of pruning downstream finetuning dataon transfer learning performance, vision transformers (ViTs) pre-trained on ImageNet21k werefine-tuned on different pruned subsets of CIFAR-10.
Pre-trained models were obtained from thetimm model library [40].
Each model was trained on a single NVIDIA TITAN
Xp 12GB graphicscard with batch size = 128, using the Adam optimizer with learning rate =
1e-5 and no weight decay.
Probe models were trained for 2 epochs each.
Pruning scores were then computed using the EL2Nsmetric [9], averaged across 10 independent random seeds.
To evaluate data pruning performance,pre-trained models were fine-tuned on each pruned dataset for 10 epochs.
To assess the effect of pruning upstream pretraining data on transfer learning performance, each ofthe ResNet-50s pre-trained on pruned subsets of ImageNet1k in Fig.
3 D was fine-tuned on all of
CIFAR-10.
Each model was trained on a single NVIDIA TITAN
Xp 12GB graphics card with batchsize = 128, using the RMSProp optimizer with learning rate = 1e-4 and no weight decay.
Probemodels were trained for 2 epochs each.
Pruning scores were then computed using the EL2Ns metric[9], averaged across 10 independent random seeds.
To evaluate data pruning performance, pre-trainedmodels were fine-tuned on each pruned dataset for 10 epochs.
C Breaking compute scaling laws via data pruning
[Figure]
s t
[Figure]
rr o r
[Figure]
T e s t
e rr o r ( t o p -% )
Epoch
Epoch that data can afford exponential savings compute and we evidence that the is true for trained on -10 and ImageNet.
We the learning experiments in Fig.
1A, keeping track
Motivated by this, we next investigate whether the convergence time of neural networks trainedon pruned datasets depends largely on the number of examples and not their difficulty, potentiallyallowing for exponential compute savings.
We investigate the learning curves of a ResNet18 trainedon CIFAR-10 and a ResNet50 on ImageNet for several different pruning fractions (Fig. 8B).
Whileprevious works have fixed the number of iterations [9], here we fix the number of epochs , so that themodel trained on 60% of the full dataset is trained for only 60% the iterations of the model trainedon the full dataset, using only 60% the compute.
Nevertheless, we find that the learning curves arestrikingly similar across pruning fractions, and appear to converge equally quickly.
These resultssuggest that data pruning could lead to large compute savings in practical settings, and in ongoingexperiments we are working to make the analogs of Fig.
8A for ResNets on CIFAR-10 and ImageNetto quantify this benefit.
D Additional scaling experiments
In Fig.
9 we perform additional scaling experiments using the EL2Ns and self-supervised prototypesmetrics.
In Fig.
10 we give a practical example of a cross over from exponential to power-law scalingwhen the probe student has limited information about the teacher (here a model trained for only asmall number of epochs on SVHN) .
E Extremal images according to different metrics
In Fig.
6, we showed extremal images for two metrics (self-supervised prototypes, memorization) anda single class.
In order to gain a better understanding of how extremal images (i.e. images that areeasiest or hardest to learn according to different metrics) look like for all metrics and more classes,we here provide additional figures.
In order to avoid cherry-picking classes while at the same timemaking sure that we are visualizing images for very different classes, we here show extreme imagesfor classes 100, ..., 500 while leaving out classes 0 and 400 (which would have been part of thevisualization) since those classes almost exclusively consist of images containing people (0: tench,400: academic gown).
The extremal images are shown in Figures 11,12,13,14,15,16,17,18.
[Figure]
s
[Figure]
p
[Figure]
t e s t
[Figure]
T o p - 5 t e s t
e rr o r ( % )
A
[Figure] [Figure]
generalization error.
h a r d r a n d o m e a s y h
a r d i n f l u e n c e m a x
e a s y h
a r d i n f l u e n c e s u m - a b s e a s y h a r d s e l f - s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d E L
[Figure]
e a s y h
a r d
DDD e a s y h a r d E L 2 N ( 2m o d e l s )
e a s y h
a r d m e m o r i z a t i o n e a s y h a r d a c t i v e l e a r n i n g e a s y h
a r d f o r g e tt i n g s h a r d r a n d o m e a s y h
a r d i n f l u e n c e m a x
e a s y h
a r d i n f l u e n c e s u m - a b s e a s y h a r d s e l f - s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d E L
[Figure]
e a s y h
a r d
DDD e a s y h a r d E L 2 N (m o d e l s )
e a s y h
a r d m e m o r i z a t i o n e a s y h a r d a c t i v e l e a r n i n g e a s y h
a r d f o r g e tt i n g h a r d r a n d o m e a s y h
a r d i n f l u e n c e m a x
e a s y h
a r d i n f l u e n c e s u m - a b s e a s y h a r d s e l f - s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d E L
[Figure] [Figure]
e a s y h
a r d
DDD e a s y h a r d E L 2 N ( 2m o d e l s )
e a s y h
a r d m e m o r i z a t i o n e a s y h a r d a c t i v e l e a r n i n g e a s y h
a r d f o r g e tt i n g e a s y h
a r d r a n d o m e a s y h a r d i n f l u e n c e m a x
e a s y h
a r d i n f l u e n c e s u m - a b s e a s y h a r d s e l f - s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d s u p e r v
i s e d p r o t o t y p
e s e a s y h
a r d E L
[Figure]
e a s y h
a r d
DDD e a s y h a r d E L 2 N ( 2m o d e l s )
e a s y h
a r d m e m o r i z a t i o n e a s y h a r d a c t i v e l e a r n i n g e a s y h
a r d f o r g e tt i n g
F Impact of number of clusters k on self-supervised prototypes
Our self-supervised prototype metric is based on k -means clustering, which has a single hyperparameter k .
By default and throughout the main paper, we set k = 1000 , corresponding to the number ofclasses in ImageNet.
Here, we investigate other settings of k to understand how this hyperparameterimpacts performance.
As can be seen in Table 1, k does indeed have an impact on performance, andvery small values for k (e.g. k < 10 ) as well as very large values for k (e.g. k = 50 , 000 ) both leadto performance impairments.
At the same time, performance is relatively high across very differentin-between settings for k .
In order to assess these results, it may be important to keep in mind that± 0 .
54% corresponds to plus/minus 2 standard deviations of performance when simply training thesame model multiple times (with different random initialization).
Overall, these results suggest that ifthe number of clusters k deviates at most by one order of magnitude from the number of classes inthe dataset (for ImageNet-1K), the exact choice of k does not matter much.
[Table]
G Impact of ensemble prototypes
The self-supervised prototypes metric is based on k -means clustering in the embedding space of a selfsupervised (=SSL) model.
Since even otherwise identical models trained with different random seedscan end up with somewhat different embedding spaces, we here investigated how the performanceof our self-supervised prototypes metric would change when averaging the scores derived from fivemodels, instead of just using a single model’s score.
The results, shown in Table 2, indicate thatensembling the self-supervised prototype scores neither improves nor hurts performance.
This isboth good and bad news: Bad news since naturally any improvement in metric development leadsto better data efficiency; on the other hand this is also good news since ensembles increase thecomputational cost of deriving the metric—and this suggests that ensembling is not necessary toachieve the performance we achieved (unlike in other methods such as ensemble active learning).
[Table]
H Relationship between pruning and class (im-)balance
Motivation.
It is well-known that strong class imbalance in a dataset is a challenge that needs tobe addressed.
In order to understand the effect of pruning on class (im-)balance, we quantified thisrelationship.
For context, if pruning according to a certain metric preferentially leads to discardingmost (or even all) images from certain classes, it is likely that the performance on those classes willdrop as a result if this imbalance is not addressed.
Class imbalance metric: pruning amplifies class imbalance.
Since a standard measure of class(im-)balance—dividing the number of images for the majority class by the number of images for theminority class—is highly sensitive to outliers and discards information about the 998 non-extreme
ImageNet classes, we instead calculated a class balance score b ∈
[0% , 100%] as the average classimbalance across any two pairs of classes by computing the expectation over taking two randomclasses, and then computing how many images the minority class has in proportion to the majorityclass.
For instance, a class balance score of 90% means that on average, when selecting two randomclasses from the dataset, the smaller of those two classes contains 90% of the number of images ofthe larger class (higher=better; 100% would be perfectly balanced).
In Fig.
19, we observe that dataset pruning strongly increases class imbalance.
This is the case bothwhen pruning away easy images and when pruning away hard images, and the effect occurs forall pruning metrics except, of course, for random pruning.
Class imbalance is well-known to be achallenge for deep learning models when not addressed properly [41].
The cause for the amplifiedclass imbalance is revealed when looking at class-conditional differences of metric scores (Figs.
21and 22):
The histograms of the class-conditional score distributions show that for many classes, most(if not all) images have very low scores, while for others most (if not all) images have very highscores.
This means that as soon as the lowest / highest scoring images are pruned, certain classes arepruned preferentially and thus class imbalance worsens.
We thus use 50% class balancing for our ImageNet experiments.
This ensures that every class has atleast 50% of the images that it would have when pruning all classes equally (essentially providinga fixed floor for the minimum number of images per class).
This simple fix is an important step toaddress and counteract class imbalance, although other means could be used as well; and ultimatelyone would likely want to use a self-supervised version of class (or cluster) balancing when it comesto pruning large-scale unlabeled datasets.
For comparison purposes, the results for ImageNet pruningwithout class balancing are shown in supplementary Fig.
[Figure]
a
[Figure]
(c) influence sum
[Figure]
abs
[Figure]
[Figure] [Figure]
) supervised
[Figure]
f)
[Figure]
(1 model)
[Figure]
l
[Figure] [Figure]
[Figure] [Figure]
20 models
[Figure] [Figure]
C l a ss b a l a n c e
[Figure]
% ) (i) memorization
[Figure] [Figure]
[Figure]
C l a ss b a l a n c e ( % )
[Figure]
[Figure]
0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.
r a n d o m 0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.0 0.
i n f l u e n c e m a x 0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.0 0.
i n f l u e n c e s u m - a b s 0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.
s
e l f - s u p e
r
v i s e d p r o t o t y p
e s 0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.
s u
p e r
v
i s e d p r o t o t y p
e s 0102030405060708090100110120130140150160170180190200210220230240250260270280290300310320330340350360370380390400410420430440450460470480490500510520530540550560570580590600610620630640650660670680690700710720730740750760770780790800810820830840850860870880890900910920930940950960970980990 Class rank 0.0 0.
E L 2 N ( 1 m o d e l )
[Figure]
DDD
[Figure]
E L 2 N ( 2 0 m o d e l s )
[Figure]
m e m o r
i z a t
i o n
[Figure]
a c t i v e l e a r n i n g
[Figure]
f o r g e tt i n g
I Effect of pruning on class-conditional accuracy and fairness
In order to study the effect of dataset pruning on model fairness, at least with respect to specific
ImageNet classes, we compared the class-conditional accuracy of a ResNet-50 model trained on thefull ImageNet dataset, versus that of the same model trained on an 80% subset obtained after pruning.
We used two supervised pruning metrics (EL2N, memorization) and one self-supervised pruningmetric (self-supervised prototypes) for obtaining the pruned dataset.
In all three cases, and across all1000 classes, we found that the class-conditional accuracy of the model trained on a pruned subset ofthe dataset remains quite similar to that of the model trained on the full dataset (Fig. 23).
However,we did notice a very small reduction in class-conditioned accuracy for ImageNet classes that wereleast accurately predicted by models trained on the entire dataset (blue lines slightly above red unitylines when class-conditioned accuracy is low).
This suggests that pruning yields a slight systematicreduction in the accuracy of harder classes, relative to easier classes, though the effect is small.
While we have focused on fairness with respect to individual ImageNet classes, any ultimate testof model fairness should be conducted in scenarios that are specific to the use case of the deployedmodel.
Our examination of the fairness of pruning with respect to individual ImageNet classesconstitutes only an initial foray into an exploration of fairness, given the absence of any specificdeployment scenario for our current models other than testing them on ImageNet.
We leave a fullexploration of fairness in other deployment settings for future work.
[Figure]
J Interaction between data pruning and training duration
Throughout the main paper, our ImageNet experiments are based on a setting where the numberof training epochs is kept constant (i.e. we train the same number of epochs on the smaller pruneddataset as on the larger original dataset).
This means that data pruning directly reduces the number ofiterations required to train the model specifically by reducing the size of the dataset.
However, thissimultaneously places two constraints on model performance: not only training on a smaller data set,but also training for fewer iterations.
We therefore investigate how model performance changes if we train longer, as quantified by amatched iterations factor .
A matched iterations factor of 0 corresponds to the default setting used inthe paper of training for the same number of epochs (so that a smaller dataset means proportionallyfewer training iterations).
In contrast a matched iterations factor of 1 corresponds to training on thesmaller pruned dataset for a number of iterations equal to that when training on the larger initialdataset (e.g. when pruning away 50% of the dataset one would train twice as long to match thenumber of iterations of a model trained on 100% of the dataset).
Otherwise the matched iterationsfactor reflects a linear interpolation in the number of training iterations as the factor varies between 0and 1 .
The results are shown in Table 3 and indicate that training longer does indeed improve performanceslightly; however, a matched iterations factor of around 0.4–0.6 may already be sufficient to reap thefull benefit.
Any matched iterations factor strictly smaller than 1.0 comes with reduced training timecompared to training a model on the full dataset.
[Table]
K Out-of-distribution (OOD) analysis of dataset pruning
Pruning changes the data regime that a model is exposed to.
Therefore, a natural question is howthis might affect desirable properties beyond IID performance like fairness (see Appendix I) andout-of-distribution, or OOD, performance which we investigate here.
To this end, we use the model-vshuman toolbox [42] based on data and analyses from [43, 44, 45, 46, 47].
This toolbox is comprisedof 17 different OOD datasets, including many image distortions and style changes.
In Figure 24a, OOD accuracies averaged across those 17 datasets are shown for a total of 12 models.
These models all have a ResNet-50 architecture [48].
Two baseline models are trained on the full
ImageNet training dataset, one using torchvision (purple) and the other using VISSL (blue).
Humanclassification data is shown as an additional reference in red.
The remaining 10 models are VISSLtrained on pruned versions of ImageNet using pruning fractions in {0.1, 0.2, 0.3, 0.4, 0.5} and ourself-supervised prototype metric.
A pruning fraction of 0.3 would correspond to “fraction of datakept = 0.7”, i.e. to training on 70% of ImageNet while discarding the other 30%.
We investigated twodifferent settings: discarding easy examples (the default used throughout the paper), which is denotedas “Best Case” (or BC) in the plots; and the reverse setting, i.e. discarding hard examples denotedas Worst Case, or WC.
(These terms should be taken with a grain of salt; examples are only insofarbest- or worst case examples as predicted by the metric, which itself is in all likelihood far less thanperfect.)
The results are as follows:
In terms of OOD accuracy (Figure 24a), best-case pruning in greenachieves very similar accuracies to the most relevant baseline, the blue ResNet-50 model trainedvia VISSL.
This is interesting since oftentimes, OOD accuracies closely follow IID accuraciesexcept for a constant offset [49], and we know from Figure 5 that the self-supervised prototypemetric has a drastic performance impairment when pruning away 40% of the data, yet the model“BC_pruning-fraction-0.4” still achieves almost the same OOD accuracy as the baseline trained onthe full dataset.
The core take-away is: While more analyses would be necessary to investigatewhether pruning indeed consistently helps on OOD performance, it seems safe to concludethat it does not hurt OOD performance on the investigated datasets compared to an accuracymatched baseline.
(The control setting, pruning away hard examples shown in orange, leads to muchlower IID accuracies and consequently also lower OOD accuracies.)
For reference, the numericalresults from Figure 24a are also shown in Table 4.
Figures 24b, 24c and 24d focus on a related question, the question of whether models show humanlike behavior on OOD datasets.
Figure 24b shows that two models pruned using our self-supervisedprototype metric somewhat more closely match human accuracies compared to the baseline in blue;
Figures 24c and 24d specifically focus on image-level consistency with human responses.
In termsof overall consistency (c), the baseline scores best; in terms of error consistency pruned modelsoutperform the VISSL-trained baseline.
For details on the metrics we kindly refer the interestedreader to [42].
Numerical results are again also shown in a Table (Table 5).
Finally, in Figure 25 we observe that best-case pruning leads to slightly higher shape bias as indicatedby green vertical lines plotting the average shape bias across categories, which are shifted to the leftof the baseline; while worst-case pruning in orange is shifted to the right.
An outlier is the torchvisiontrained model in purple with a very strong texture bias; we attribute this to data augmentationdifferences between VISSL and torchvision training.0.
OO D a cc u r a c y
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( t o r c h v
i s i o n )
B C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( V I
SS L ) hu
m a n s
[Table]
p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( t o r c h v
i s i o n )
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( V I
SS L )
B C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 .hu
m a n s n
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( t o r c h v
i s i o n )
B C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( V I
SS L )
hu
m a n
s e rr o r c
o n s i s t e n c y
W C _ p r un i n g - f r a c t i o n - 0 . W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( V I
SS L )
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 . B C _ p r un i n g - f r a c t i o n - 0 .
W C _ p r un i n g - f r a c t i o n - 0 .
B C _ p r un i n g - f r a c t i o n - 0 .
R e s N e t -
( t o r c h v
i s i o n )
hu
m a n s
[Figure]
Two baseline models are trained on the full ImageNet training
, one torchvision ( ) and the other
VISSL ( Human comparison is shown
[Table]
[Table]
Fraction of 'shape' decisions
Fraction
of 'texture' decisions
S h a p e c a t e g
o
r
i e s