Optimizing Data Collection for Machine Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect.
Over-collecting data incurs unnecessary present costs, while under-collecting mayincur future costs and delay workflows.
We propose a new paradigm for modelingthe data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, andpenalties for failing to meet the targets.
Additionally, this formulation generalizesto tasks requiring multiple data sources, such as labeled and unlabeled data usedin semi-supervised learning.
To solve our problem, we develop Learn-OptimizeCollect (LOC), which minimizes expected future collection costs.
Finally, wenumerically compare our framework to the conventional baseline of estimating datarequirements by extrapolating from neural scaling laws.
We significantly reducethe risks of failing to meet desired performance targets on several classification,segmentation, and detection tasks, while maintaining low total collection costs.
Introduction
When deploying a deep learning model in an industrial application, designers often mandate that themodel must meet a pre-determined baseline performance, such as a target metric over a validationdata set.
For example, an object detector may require a certain minimum mean average precisionbefore being deployed in a safety-critical setting.
One of the most effective ways of meeting targetperformances is by collecting more training data for a given model.
Determining how much data is needed to meet performance targets can impact costs and developmentdelays.
Overestimating the data requirement incurs excess costs from collection, cleaning, andannotation.
For instance, annotating segmentation masks for a driving data set takes between 15to 40 seconds per object.
For 100 , 000 images the annotation could require between 170 and 460days-equivalent of time [1, 2].
On the other hand, collecting too little data may incur future costs andworkflow delays from having to collect more later.
For example, in medical imaging applications,this means further clinical data acquisition rounds that require expensive clinician time.
In the worstcase, designers may even realize that a project is infeasible only after collecting insufficient data.
The growing literature on sample complexity in machine learning has identified neural scaling lawsthat scale model performance with data set sizes according to power laws [3–10].
For instance, Rosenfeld et al.
[6] fit power law functions on the performance statistics of small data sets to extrapolate thelearning curve with more data.
In contrast, Mahmood et al.
[2] consider estimating data requirementsand show that even small errors in a power law model of the learning curve can translate to massivelyover- or underestimating how much data is needed.
Beyond this, different data sources have differentcosts and scale differently with performance [11].
For example, although unlabeled data may be easierto collect than labeled data, some semi-supervised learning tasks may need an order of magnitudemore unlabeled data to match the performance of a small labeled set.
Thus, collecting more databased only on estimation will fail to capture uncertainty and collection costs
[Figure]
Collect
Our contributions are as follows.
(1) We propose the optimal data collection problem in machine learning, which formalizes data collection workflows.
(2) We introduce Learn-Optimize-Collect
(LOC), a learning-and-optimizing framework that minimizes future collection costs, can be solved via gradient descent and has analytic solutions in some settings.
(3) We generalize the data collection problem and LOC to a multi-variate setting where different types of data have different costs.
To thebest of our knowledge, this is the first exploration of data collection with general multiple data setsin machine learning, covering for example, semi-supervised and long-tail learning.
(4) We performexperiments over classification, segmentation, and detection tasks to show, on average, approximatelya 2 × reduction in the chances of failing to meet performance targets, versus estimation baselines.
Related work
Neural Scaling Laws.
According to the neural scaling law literature, the performance of a modelon a validation set scales with the size of the training data set q via a power law V ∝ θ 0 q θ 1 [5, 6, 8–10, 12–16].
Hestness et al.
[5] observe this property over vision, language, and audio tasks, Bahri et al.[9] develop a theoretical relationship under assumptions on over-parametrization and the Lipschitzcontinuity of the loss, model, and data, and Rosenfeld et al.
[6] estimate power laws using smallerdata sets and models to extrapolate future performance.
Multi-variate scaling laws have also beenconsidered for some specific tasks, for example in transfer learning from synthetic to real datasets [11].
Finally, Mahmood et al.
[2] explore data collection by estimating the minimum amountof data needed to meet a given target performance over multiple rounds.
Our paper extends theseprior studies by developing an optimization problem to minimize the expected total cost of datacollected.
Specifically, we incorporate the uncertainty in any regression estimate of data requirementsand further generalize to multiple data sources with different costs.
Active Learning.
In active learning, a model sequentially collects data by selecting new subsets ofan unlabeled data pool to label under a pre-determined labeling budget that replenishes after eachround [17–21].
In contrast, our work focuses on systematically determining an optimal collectionbudget. After determining how much data to collect, we can use active learning techniques to collectthe desired amount of data.
Statistical Learning Theory.
Theoretical analysis of the sample complexity of machine learningmodels is typically only tight asymptotically, but some recent work have empirically analyzed neural networks, provide empirical validation, and suggest using them to estimate data requirements.
Incontrast, our paper formally explores the consequences of collection costs on data requirements.
Optimal Experiment Design.
The topic of how to collect data, select samples, and design scientificexperiments or controlled trials is well-studied in econometrics [24–26].
For example, Bertsimaset al.
[27] optimize the assignment of samples into control and trial groups to minimize inter-groupvariances.
Most recently, Carneiro et al.
[28] optimize how many samples and covariates to collect ina statistical experiment by minimizing a treatment effect estimation error or maximizing t -test power.
However, our focus on industrial machine learning applications differs from experiment design byhaving target performance metrics and continual rounds of collection and modeling.
Main Problem
In this section, we give a motivating example before introducing the formal data collection problem.
We include a table of notation in Appendix ?? .
Motivating Example.
A startup is developing an object detector for use in autonomous vehicleswithin the next T = 5 years.
Their model must achieve a mean Average Precision greater than V ∗ =95% on a pre-determined validation set or else they will lose an expected profit of P = $1 , 000 , 000 .
Collecting training data requires employing drivers to record videos and annotators to label the data,where the marginal cost of obtaining each image is approximately c = $1 .
In order to manage annualfinances, the startup must plan how much data to collect at the beginning of each year.
Let z ∼ p ( z ) be data drawn from a distribution p .
For instance, z :=
( x, y ) may correspond toimages x and labels y .
Consider a prediction problem for which we train a model with a data set Dof points sampled from p ( z ) .
Let V ( D ) be a score function evaluating the model trained on D .
Optimal Data Collection.
We possess an initial data set D q 0 :=
{ z i } q 0 i =1 of q 0 points; we omit thesubscript on D referring to its size when it is obvious.
Our problem is defined by a target score
V ∗ >
V ( D q 0 ) , a cost-per-sample c of collection, a horizon of T rounds, and a penalty P for failure.
At the end of each round t ∈ { 1 , . . .
, T } , let q t be the current amount of data collected.
Our goal isto minimize the total cost of collection while building a model that can achieve the target score:
[Equation]
[Equation]
We collect training data iteratively over multiple rounds (see Figure 1), where in each round, we1.
Decide to grow the data set to q t ≥ q t − 1 points by sampling ˆ D :=
{ ˆ z i } q t − q t − 1 i = ∼ p ( z ) .
Paya cost c ( q t − q t − 1 ) and update D ← D ∪ ˆ D .2.
Train the model and evaluate the score.
If V ( D ) ≥ V ∗ , then terminate.
If t = T , then
pay the penalty P and terminate.
Otherwise, repeat for the next round.
The model score typically increases monotonically with data set size [5, 6].
This means that theminimum cost strategy for (1) is to collect just enough data such that V ( D q T ) =
V ∗ .
We canestimate this minimum data requirement by modeling the score function as a stochastic process.
Let
V q :=
V ( D q ) and let { V q } q ∈ Z
+ be a stochastic process whose indices represent training set sizesin different rounds.
Then, collecting data in each round yields a sequence of subsampled data sets
D q t − 1 ⊂ D q t and their performances V ( D q t ) .
The minimum data requirement is the stopping time
[Equation]
which is a random variable giving the first time that we pass the target.
Note that q ∗ 1 = · · · = q ∗ T =
D ∗is a minimum cost solution to the optimal data collection problem, incurring a total cost c ( D ∗ − q 0 ) 1 .
Estimating D ∗ using past observations of the learning curve is difficult since we have only T rounds.
Further, Mahmood et al.
[2] empirically show that small errors in fitting the learning curve can causemassive over- or under-collection.
Thus, robust policies must capture the uncertainty of estimation.
Learn-Optimize-Collect (LOC)
Our solution approach, which we refer to as Learn-Optimize-Collect (LOC), minimizes the totalcollection cost while incorporating the uncertainty of estimating D ∗ .
Although D ∗ is a discreterandom variable, it is realized typically on the order of thousands or greater.
To simplify our problemand ensure differentiability, we assume that D ∗ is continuous and has a well-defined density.
Assumption 1.
The random variable D ∗ is absolutely continuous and has a cumulative densityfunction (CDF) F ( q ) and probability density function (PDF) f ( q ) :
= dF ( q ) /dq .
In Section 4.1, we first develop an optimization model when given access to the CDF f ( q ) and PDF
F ( q ) .
In Section 4.2, we estimate these distributions and combine them with the optimization model.
Finally in Section 4.3, we delineate our optimization approach from prior regression methods.
4.1 Optimization Model
We first propose an optimization problem that at any given round t can simultaneously solve for theoptimal amounts of data to collect q t , . . .
, q T in all future rounds.
Consider the initial setting at t = 1 .
In order to develop intuition, let us first suppose that we know a priori the exact stopping time D ∗
Then, problem (1) can be re-written as
[Equation]
where the objective function is defined recursively as follows
[Equation]
The second line follows from gathering the terms.
The third line follows from observing that sinceq 1 ≤ q 2 ≤ · · · q T is a constraint, the product of the indicators is equal to the maximum.
In practice, we do not know D ∗ a priori since it is an unobserved random variable.
Instead, suppose wehave access to the CDF F ( q ) .
Then, we take the expectation over the objective E [ L ( q 1 , . . .
, q T ; D ∗ )]to formulate a stochastic optimization problem for determining how much data to collect:
[Equation]
Note that the collection variables should be discrete q 1 , . .
, q T ∈
Z
+ , but similar to the modelingof D ∗ , we relax the integrality requirement, optimize over continuous variables, and round the finalsolutions.
Furthermore, although problem (4) is constrained, we can re-formulate it with variablesd t :=
q t − q t − 1 ; this consequently replaces the current constraints with only non-negativity constraintsd t ≥ 0 .
Finally due to Assumption 1, problem (4) can be optimized via gradient descent.
4.2 Learning and Optimizing the Data Requirement
Solving problem (4) requires access to the true distribution F ( q ) , which we do not have in reality.
Ineach round, given a current training data set D q t of q t points, we must estimate these distributionfunctions F ( q ) and f ( q ) and then incorporate them into our optimization problem.
Given a current data set D q t , we may sample an increasing sequence of R subsets D q t /R ⊂ D 2 q t /R ⊂ 152 · · · ⊂ D q t , fit our model to each subset, and compute the scores to obtain a data set of the learningcurve R := { ( rq t /R, V
( D rq t /R ))
Rr =1 .
In order to model the distribution of D ∗ , we can take Bbootstrap resamples of R to fit a series of regression functions and obtain corresponding estimates
{ ˆ D b } Bb =1 .
Given a set of estimates of the data requirement, we estimate the PDF via Kernel Density Estimation (KDE).
Finally to fit the CDF, we numerically integrate the PDF.
In our complete framework, LOC, we first estimate F ( q ) and f ( q ) .
We then use these models to solveproblem (4).
Note that in the t -th round of collection, we fix the prior decision variables q 1 , . . .
q t − 1constant.
Finally, we collect data as determined by the optimal solution q ∗ t to problem (4).
Full detailsof the learning and optimization steps, including the complete Algorithm, are in Appendix ?? .
3 Comparison to Mahmood et al.
Our prediction model extends the previous approach of Mahmood et al.
[2], who consider onlypoint estimation of D ∗ .
They (i) build the set R , (ii) fit a parametric function ˆ v ( q ; θ ) to R vialeast-squares minimization, and (iii) solve for ˆ D
= arg
min q { q | ˆ v ( q ; θ ) ≥ V ∗ } .
They useseveral parametric functions from the neural scaling law literature, including the power law function,ˆ v ( q ; θ ) :
= θ 0 q θ 1 + θ 2 [8, 2], and use an ad hoc correction factor obtained by trial and error onpast tasks to help decrease the failure rate.
Instead, we take bootstrap samples of R to fit multipleregression functions, estimate a distribution for ˆ D , and incorporate them into our novel optimizationmodel.
Finally, we show in the next two sections that our optimization problem has analytic solutionsand extends to multiple sources.
Analytic Solutions for the T = 1 Setting
In this section, we explore analytic solutions for problem (4).
The unobservable D ∗ and sequentialdecision-making nature suggest this problem can be formulated as a Partially Observable Markov
Decision Process (POMDP) with an infinite state and action space (see Appendix ?? ), but suchproblems rarely permit exact solution methods [29].
Nonetheless, we can derive exact solutions forthe simple case of a single T = 1 round, re-stated below
[Equation]
Theorem 1.
Assume F ( q ) is strictly increasing and continuous.
For any ϵ such that F ( q 0 ) < 1 − ϵ , let
P :=
c/f ( F − 1 (1 − ϵ )) .
The optimal solution to the corresponding problem (5) is q ∗ 1 = F − 1 (1 − ϵ ) .
Furthermore, this solution satisfies F ( q ∗ 1 ) = 1 − ϵ .
When the penalty P is specified via a failure risk ϵ , the optimal solution to problem (5) is equal to aquantile of the distribution of D ∗ .
We defer the proof and some auxiliary results to Appendix ?? .
Theorem 1 further provides guidelines on choosing values for the cost and penalty parameters.
Whilec is the dollar-value cost per-sample, which includes acquisition, cleaning, and annotation, P canreflect their inherent regret or opportunity cost of failing to meet their target score.
A designer canaccept a risk ϵ of failing to collect enough data Pr { q ∗ < D ∗ } = ϵ .
From Theorem 1, their optimalstrategy should be to collect F − 1 (1 − ϵ ) points, which is also the optimal solution to problem (5).
The Multi-variate LOC: Collecting Data from Multiple Sources
So far, we have assumed that a designer only chooses how much data to collect and must pay afixed per-sample collection cost.
We now explore the multi-variate extension of the data collectionproblem where there are different types of data with different costs.
For example, consider long-taillearning where samples for some rare classes are harder to obtain and thus, more expensive [30],semi-supervised learning where labeling data may cost more than collecting unlabeled data [31], ordomain adaptation where a source data set is easier to obtain than a target set [32].
In this section, wehighlight our main formulation and defer the complete multi-variate LOC to Appendix ?? .
Consider K ∈ N data sources (e.g., K = 2 with labeled and unlabeled) and for each k ∈ { 1 , . .
, K } ,let z k ∼ p
k
( z k ) be data drawn from their distribution.
We train a model with a data set D := ∪ Kk =1 D kwhere each D k contains points of the k -th source.
The performance or score function of our model is
V ( D 1 , . . .
, D K ) .
For each k , we initialize with q k 0 points.
Let q 0 = ( q 10 , . .
, q K 0 ) T denote the vectorof data set sizes and let c = ( c 1 , . . .
, c K ) T denote costs (i.e., c k is the cost of collecting data from p k
( z k ) ).
Given a target V ∗ , penalty P , and T rounds, we want to minimize the total cost of collection
[Equation]
[Equation] [Equation]
We can follow the same steps shown in Section 4 to solve this problem.
First, the learning curve isnow a stochastic process { V q } q ∈ Z K + indexed in K dimensions.
Further, the multi-variate analogue ofthe minimum data requirement in (2) is now a vector that states the minimum cost amount of dataneeded to meet the target score:
[Equation]
We randomly pick a unique solution to break ties.
From Assumption 1, D ∗ is a random vector witha PDF f ( q ) and a CDF F ( q ) :
= (cid:82) q 0 f (ˆ q ) d
ˆ q .
Finally, the multi-variate analogue of the stochasticproblem (4) is
[Equation]
The Multi-variate LOC requires multi-variate PDFs, which we can fit in the same way as discussedin Section 4.2.
However, we now need multi-variate regression functions that can accommodatedifferent types of data.
In Appendix ?? , we propose an additive family of power law regressionfunctions that can handle an arbitrary number of K sources.
In our experiments, we also generalizethe estimation approach of Mahmood et al.
[2] to the multi-source setting for comparison.
Empirical Results
We explore the data collection problem over two sets of experiments covering single-variate K = 1(Section 4) and multi-variate K =
(Section 6) problems.
We consider image classification,segmentation, and object detection tasks.
For every data set and task, LOC significantly reduces thenumber of instances where we fail to meet a data requirement V ∗ , while incurring a competitive costwith respect to the conventional baseline of naïvely estimating the data requirement [2].
In this section, we summarize the main results.
We detail our data collection and experiment setup in
Appendix ?? . We expand our full results and experiments with additional baselines in Appendix ?? .
7.1 Data and Methods
When K = 1 , the designer decides how much data to sample without controlling the type ofdata.
We explore classification on CIFAR-10 [33], CIFAR-100 [33], and ImageNet [34], where wetrain ResNets [35] to meet a target validation accuracy.
We explore semantic segmentation using
Deeplabv3 [36] on BDD100K [37], which is a large-scale driving data set, as well as Bird’s-Eye-View(BEV) segmentation on nuScenes [38] using the ‘Lift Splat’ architecture [39]; for both tasks, wedesire a target mean intersection-over-union (IoU).
We explore 2-D object detection on PASCAL
VOC [40, 41] using SSD300 [42], where we evaluate mean average precision (mAP).
When K = 2 , the designer collects two types of data with different costs.
We first divide CIFAR-100into two subsets containing data from the first and last 50 classes, respectively.
Here, we assumethat the first 50 classes are more expensive to collect than the last; this mimics a real-world scenariowhere collecting data for some classes (e.g., long-tail) is more expensive than others.
We then exploresemi-supervised learning on BDD100K where the labeled subset of this data is more expensive thanthe unlabeled data; the cost difference between these two types is equal to the cost of data annotation.
We use a simulation model of the deep learning workflow following the procedure of Mahmoodet al.
[2], to approximate the true problem while simplifying the experiments (see Appendix ?? forfull details).
To avoid repeatedly sampling data, re-training a model, and evaluating the score, eachsimulation uses a piecewise-linear approximation of a ‘ground truth’ learning curve that returnsmodel performance as a function of data set size.
In our problems, we initialize with q 0 = 10% of thefull data set (we use 20% for VOC).
Then in each round, we solve for the amount of data to collectand then call the piecewise-linear learning curve to obtain the current score.
[Figure]
o f p o i n t s q * / D *
[Figure]
(T
[Figure] [Figure]
R a t i o o f p o i n t s q * / D *
[Figure]
(T
[Figure] [Figure]
26 28 30 V * 0.
R a t i o o f p o i n t s q * / D *
[Figure]
T
[Figure] [Figure]
R a t i o o f p o i n t s q * / D *
[Figure]
requirement.
LOC almost always remains slightly above the black line, meaning we rarely fail to meet the target.
[Table] [Table]
and failure rate measured over a range of V ∗ for each T and data set.
We fix c = 1 and P = 10 7 ( P = 10 6 for VOC and
P = 10 8 for ImageNet).
The best performing failure rate for each setting is bolded.
The cost ratio is measured only for instances that achieve V ∗ .
LOC consistently reduces the average failure rate, often down to 0% , while keeping the average cost ratio almost always below 1 (i.e., spending at most 2 × the optimal amount).
We compare LOC against the conventional estimation approach of Mahmood et al.
[2], who fit aregression model to the learning curve statistics, extrapolate the learning curve for larger data sets,and then solve for the minimum data requirement under this extrapolation.
There are many differentregression models that can be used to fit learning curves [12, 14, 5, 8], but power laws are the mostcommonly studied approach in the neural scaling law literature.
Consequently, we use power lawregression to model the learning curve for the baseline and LOC.
7.2 Main Results
We consider T = 1 , 3 , 5 rounds and V ∗ ∈
[ V ( D q 0 )
+ 1 , V ( D )]
targets, where D is the entire dataset.
We evaluate all methods on (i) the failure rate, which is how often the method fails to achieve thegiven V ∗ within T rounds, and (ii) the cost ratio, which is equal to c T ( q ∗ T − q 0 ) /
c T ( D ∗ − q 0 ) − 1 .
For K = 1 , we also measure the ratio of points collected q ∗ T /D ∗ .
Although there is a natural trade-offbetween low cost ratio (under-collecting) and failure rate (over-collecting), we emphasize that ourgoal is to have low cost but with zero chance of failure.
The Value of Optimization over Estimation when K = 1 .
Figure 2 compares LOC versus thecorresponding power law regression baseline when c = 1 and P = 10 7 ( P = 10 6 for VOC and
P = 10 8 for ImageNet).
If a curve is below the black line, then it failed to collect enough data tomeet the target.
LOC consistently remains above this black line for most settings.
In contrast, evenwith up to T = 5 rounds, collecting data based only on regression estimates leads to failure.
[Figure] [Figure]
Table 1 aggregates the failure rates and cost ratios for each setting.
To summarize, LOC fails at lessthan 10% of instances for 12 / 18 settings, whereas regression fails over 30% for 15 / 18 settings.
Inparticular, regression nearly always under-collects data when given a single T = 1 round.
Here, LOCreduces the risk of under-collecting by 40% to 90% over the baseline.
While this leads to a marginalincrease in costs, our cost ratios are consistently less than 0 .
5 for 12 / 18 settings, meaning that wespend at most 50% more than the true minimum cost.
We remark that previously, Mahmood et al.
[2] observed that incorrect regression estimates necessitated real machine learning workflows to collect data over multiple rounds.
Instead, with LOC, wecan make significantly improved data collection decisions even with a single round.
Robustness to Cost and Penalty Parameters (see Appendix ?? for details).
Figure 3 evaluates theratio of points collected for T = 5 when the cost and the penalty of the optimization problem arevaried.
Our algorithm is robust to variations in these parameters, as LOC retain the same shape andscale for almost every parameter setting and data set.
Further, LOC consistently remain above thehorizontal 1 line, showing that even after varying c and P , we do not fail as frequently as the baseline.
Finally, validating Theorem 1, the penalty parameter P provides natural control over the amount ofdata collected.
As we increase P , the ratio of data collected increases consistently.
The Value of Optimization over Estimation when K = 2 (Appendix ??).
Figure 4 compares
LOC versus regression at T = 5 with different costs, showing that we maintain a similar cost ratio tothe regression alternative, but with lower failure rates.
Table 2 aggregates failure rates and cost ratiosfor all settings, showing LOC consistently achieves lower failure rates for nearly all settings of T .
When T = 5 , LOC also achieves lower cost ratios versus regression on CIFAR-100, meaning that
[Table] [Table]
and failure rate over different
V ∗ for each T and c , after removing 99 -th percentile outliers.
We fix P = 10 13 for
CIFAR-100 and P = 10 8 for
BDD100K.
The best performing failure rate for each setting is bolded.
The cost ratio is measured over instances that achieve V ∗ .
LOC consistently reduces the average failure rate, and for T > 1 , preserves the cost ratio.
Further, LOC is more robust to uneven costs than regression.
with multiple rounds of collection, we can ensure meeting performance requirements while payingnearly the optimal amount of data.
However, solving the optimization problem is generally moredifficult as K increases, and we sometimes over-collect data by large margins; consequently, wereport these results after removing the 99 -th percentile outliers with respect to total cost for bothmethods.
Nonetheless, this challenge remains when T
= 1 , particularly for CIFAR-100.
Discussion
We develop a rigorous framework for optimizing data collection workflows in machine learningapplications, by introducing an optimal data collection problem that captures the uncertainty inestimating data requirements.
We generalize this problem to more realistic settings where multipledata sources incur varying costs of collection.
We validate our solution algorithm, LOC, on six datasets covering classification, segmentation, and detection tasks to show that we consistently meetpre-determined performance metrics regardless of costs and time horizons.
Our approach relies on estimating the CDF and PDF of the minimum data requirement, which isa challenging problem, especially with multiple data sources.
Nonetheless, LOC can be deployedon top of future advances in estimating neural scaling laws.
Further, we allow practitioners to inputproblem-specific costs and penalties, but these quantities may not always be readily available.
Weprovide some theoretical insight into parameter selection and show that LOC is robust to theseparameters.
Finally, our empirical analysis focuses on computer vision, but we expect our approachto be viable in other domains governed by scaling laws.
Improving data collection practices yields potentially positive and negative societal impacts.
LOCreduces the collection of extraneous data, which can, in turn, reduce the environmental costs oftraining models.
On the other hand, equitable data collection should also be considered in real-worlddata collection practices that involve humans.
We envision a potential future work to incorporateprivacy and fairness constraints to prevent over- or under-sampling of protected groups.
Finally,our method is guided by a score function on a held-out validation set.
Biases in this set may beexacerbated when optimizing data collection to meet target performance.
There is a folklore observation that over 80% of industry machine learning projects fail to reachproduction, often due to insufficient, noisy, or inappropriate data [43, 44].
Our experiments verifythis by showing that naïvely estimating data requirements will often yield failures to meet targetperformances.
We believe that robust data collection policies obtained via LOC can reduce failureswhile further guiding practitioners on how to manage both costs and time.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the Conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See theConclusion.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assump-
tion 1 and the theorem statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix ?? .
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] The code isproprietary.
The data is publicly available.
See Algorithm ?? and Appendix ?? for
implementation details.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix ?? and ?? for details.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] All experiments were over five seeds.
See the for standarddeviation ranges.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix ?? .
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all data sets
and model architectures in Appendix ?? .
(b) Did you mention the license of the assets?
[N/A] All data sets and models are publicly available.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]