Striving for data-model efﬁciency: Identifying data externalities on group performance
Abstract
Building trustworthy, effective, and responsible machine learning systems hinges on understanding how differences in training data and modeling decisions interact to impact predictive performance.
In this work, we seek to better understand how we might characterize, detect, and design for data-model synergies.
We focus on a particular type of data-model inefﬁciency, in which adding training data from some sources can actually lower performance evaluated on key sub-groups of the population, a phenomenon we refer to as negative data externalities on group performance .
Such externalities can arise in standard learning settings and can manifest differently depending on conditions between training set size and model size.
Data externalities directly imply a lower bound on feasible model improvements, yet improving models efﬁciently requires understanding the underlying data-model tensions.
From a broader perspective, our results indicate that data-efﬁciency is a key component of both accurate and trustworthy machine learning.
Introduction
Although key aspects of trustworthiness and responsibility in machine learning are often framed from an algorithmic perspective we explore an alternative framing that focuses on how our chosen modeling and training procedures perform under different data (collection) regimes.
We refer to the guiding goal of using the available training data to achieve the best possible performance for all target populations as “data efﬁciency.”
While this has clear alignment with accuracy maximization, data minimization, and fairness, it is not always clear how to test or design for data-efﬁciency generally.
We focus on a speciﬁc type of data inefﬁciency , in which adding training data from certain data sources can actually decrease performance on key groups of the target population.
We call this phenomenon negative data externalities on group performance (deﬁned formally in deﬁnition 1).
While recent works have examined how the amount of training data from one group affects performance evaluated on that same group [7, 11, 19], studying between-group trends can evidence tensions between different aspects of model performance that might otherwise go unnoticed.
In this work we formalize data externalities and their immediate cost to model performance (§2,§3).
We describe mechanisms by which negative data externalities can arise and detail why understanding these mechanisms matters for designing effective model improvements (§4).
Our experiments show how data externalities shift with conditions on sample size and model capacity
Altogether, our results suggest that data externalities may be suprisingly pervasive in real-world settings, yet avoidable by designing training procedures to promote data-efﬁciency
Our ﬁndings have implications to fairness, privacy, participation and transparency in machine learning
Related Work.
Alongside a recognition of the role of training data in achieving high-performing models [1, 19, 20] has come an appreciation that dataset size does not imply requisite quality [4], representativeness [6], or diversity [3].
For example, concerns have been raised about the use of large, multi-source text datasets, including transparency of data provenance [10] contributed to by a “documentation debt” [3].
In this work, we study how training data from different sources can affect model performance, focusing on possible externalities to key subgroups of the target population.
Prior work has proposed scaling laws to describe how the total number of training samples and model size affect aggregate performance [2, 16].
Our study connects to a growing line of research on how varying the amount of training data from different sources, groups, or classes impacts model performance, including subgroup and average accuracies [13, 19], and statistical fairness [7, 11].
Scaling laws that have been proposed to model the impact of data from one dataset (e.g. [2, 16]) or multiple source groups (e.g. [7, 19]) on model performance typically implicitly encode that more data never worsens performance, and thus do not allow for data externalities of the type we study here.
Data valuation techniques have been proposed to quantify the (possibly negative) value of individual data points [12, 18].
In a framework more similar to our own, [15] studies scenarios where leaving out entire classes from a training dataset can improve transfer learning performance.
Finally, we note that the general terms “data externalities” and “information externalities” have also been used to describe externalities on other factors, such as privacy [8] and data valuation [14].
Setting and notation
We assume that each training instance is collected from exactly one of g ∈ G source distinct source groups.
Each evaluation instance can be associated with any subset of predeﬁned evaluation groups g ∈ G eval , which may be intersecting.
In this work, we assume that groups are known at training and evaluation time.
We will often make the simplifying assumption that G eval =
G source referring to the set of groups simply as G .
We assume that one can sample instances ( x, y ) ∼ D g for each group in g ∈ G source ∪ G eval , where
D g denotes the data distribution corresponding to group g .
We say that a model f ( · ) results in group risks, or expected losses of R ( g ) =
E ( x,y )
∼D g
[ (cid:96) ( f ( x ) , y )] deﬁned with respect to to loss function (cid:96) .
Per-group risks as random variables parameterized by training dataset composition.
Following [19], we consider the learned model f θ as a random function parameterized by both the model parameters θ and allocations governing sample sizes from each source.
Denote the (random) training dataset S as the union of the |G source | sets each comprising samples from one training source:
[Equation]
where sample sizes are determined by the total dataset size n = (cid:80) g ∈G source n g and allocations n g /n .
With these deﬁnitions in place, we can deﬁne the expected risk of a training procedure as a function of the composition of the training set.
For example, for a standard training procedure that selects a model from class F Θ = { f θ } θ ∈ Θ to minimize loss (cid:96) train :
[Equation]
where θ ∗ (cid:96) train , S = arg min θ ∈ Θ (cid:96) train ( f θ , S ) .
Equation (2) gives us a framework by which to describe performance of our learning procedure across evaluation groups as sensitive to many different scenarios and choices, including the training source sample sizes n g and model class complexity.
Data externalities and their costs
We now use the framework of §2 to characterize our main phenomenon of study: when adding training data from a particular source group decreases performance for an evaluation group.
Deﬁnition 1 ( negative data externality on group risk ) .
For ﬁxed training procedure described by model class F and loss objective (cid:96) train , a negative data externality on group risk (w.r.t.
groups G eval ) occurs when the expected risk for some evaluation group can increase as a result of adding randomly sampled training data:
[Equation]
We can interpret eq.
(3) across all possible allocations that could be collected, or with respect to an existing dataset with total size n = (cid:80) g ∈G source n g , where D g denote empirical distributions and sampling is done uniformly at random.
In the second case, a data externality exposes an implicit “cost” to some evaluation group, formalized as a room for improvement, ∆ , in the following claim.
Claim 1 (Data externalities lower bound room for model improvement) .
For a training dataset with { n g } g ∈G source , for ﬁxed training procedure with model class F and training loss (cid:96) train , the maximum magnitude of data externality ∆ g eval on group g eval , ∆
[Equation]
is a lower bound on the best possible improvement in expected risk for group g eval that can be achieved using this dataset without raising the expected risk for groups disjoint from g eval .
Proof.
We can construct an alternative training procedure that ﬁrst subsets the training data uniformly at random from each source group according to the n (cid:48)
g that maximize the expression in eq.
Since groups are assumed to be known, we can selectively apply this new model to instances from g eval and use the original model for all other instances.
This split model lowers expected risk by ∆ g eval for g eval and does not alter expected risk for instances not in g eval .
As this procedure only optimizes over the training data sub-sampling, it is a lower bound for the possible performance improvements.
Claim 1 highlights that identifying data externalities can improve the model for groups on which the model under-performs, without any negative consequence to expected risk evaluated on disjoint evaluation groups.
However, data externalities also tell us something more subtle about the compatibility of our model with the underlying structures in our data.
In the next section, we investigate possible causes of data externalities, and what they mean for improving model performance.
When do negative data externalities arise?
An intuitive setting where data externalities can arise is when the complexity of model class F Θ is constrained or mis-speciﬁed so that the optimal parameters differ per group, i.e. θ ∗ g (cid:54)
= θ ∗ g (cid:48) , where θ ∗ g := arg min θ ∈ Θ E ( x,y ) ∼D g
[ (cid:96) ( f θ ( x ) , y )] .
Here we detail such a setting, as well as another example in which data externalities arise even when the optimal model is the same for all groups ( θ ∗ g = θ ∗ g (cid:48) ). [Figure] [Figure] data distributions
[Figure] The true model is shared between groups, but the distributions of features and observation noise differs. Adding small n B can detract from performance on group A due to increased noise in the training data. Our examples use an illustrative model where we assume there exists a true afﬁne relationship for each group, with shared linear weights but different intercepts (see §A.3 for exact parameters): y g = w ·
[Equation]
+
[Equation]
A · I [
[Equation] = A ] + [Equation] B · I [ g =
[Equation] ] + [Equation] g
[Equation] [Equation]
=
[Equation]
, but the model class is the set of afﬁne models shared between groups ( f θ ( x, g )
= θ 1 x + θ 2 ).
In the ﬁrst example, we vary the intercept of the true model between the groups, as well as the mean of the feature distribution (ﬁg. 1a, left).
The discrepancy between the true model and the model class results in data externalities with magnitude ( ∆ in eq.
(4)) increasing monotonically as the number of samples from the other group increases (positive slopes in ﬁg. 1a, right).
In the second example, the same model parameters apply to both groups, but we vary the spread of the feature distribution and scale of observation noise (cid:15) g between groups, effectively decreasing the signal-to-noise ratio for group B relative to group A (ﬁg. 1b, left).
This results in data externalities evaluated on group A for small to mid-range number of samples from group B , which dissipate with larger n B (ﬁg. 1b, right).
What constitutes “small to mid-range” values of n B is relative to n A : the magnitude of the negative data externalities decrease with sufﬁcient samples from group A .
In these two examples, negative data externalities arise for different reasons and the modeling intervention that would best address the data externalities depends on the cause of the tension .
In the ﬁrst example, allowing for a more complex model class which ﬁts a different intercept term for each group (expanding model capacity in a targeted way) will alleviate data externalities for any { n A , n B } .
In the second example, removing negative data externalities by splitting the model by group would eliminate the positive externalities of adding samples from group B when n B is large ( ≥ 1 e 5 in ﬁg. 1b).
A more appropriate strategy for the second example would reweight instances according to their source group in when computing and optimizing the training loss.
While data externalities signal a clear opportunity to improve performance (claim 1), the examples above highlight that best way to make model improvements will depend on the setting.
Data externalities could thus be considered as a symptom indicating sub-optimal data-efﬁciency of a given modeling procedure.
Remedying the exposed tensions in an effective manner will require understanding the underlying mechanisms giving rise to the observed data externalities.
Data externalities with real data
Experiments in this section expose negative data externalities with respect to the empirical distributions deﬁned by two different real-world datasets (see §A.1 for more details on datasets):
The Goodreads datasets [22] contain book reviews and ratings of books from different genres.
We collect data for two genres – history/biography ( history ) and fantasy/paranormal ( fantasy ) – which comprise the two groups g in our setup.
Similar to in [19], the binary prediction task is to discern whether a book review corresponds to a 5 star rating (1) or less (0), given the text of the review.
The CivilComments dataset with identity information [5] contains online comments with humanannotated labels corresponding to whether the comment is considered toxic and whether the comment is targeted at a speciﬁc group g .
We focus on the four largest identity groups present in the dataset: female, male, Christian, and Muslim (groups are determined as binary labels if the annotator average is at least 0 .
5 for that identity group, similar to toxicity labels).
Beyond evidencing that negative data externalities can manifest in real-data settings, we design experiments to understand when they manifest, in light of results from §4.
We examine how data externalities arise under different conditions on total sample size (§5.1) and model capacity (§5.2).
To identify data externalities with existing datasets, we sub-sample the available training data from each source group g uniformly at random with different allocations deﬁned by { n g } g ∈G .
To estimate eq.
(2) for each group, we measure the per-group performance of the resulting models on ﬁxed evaluation sets.
We report per-group area under the reciever operating characteristic (AUROC) as a metric that is insensitive to class imbalances (see §A.2).
We report variability across multiple random draws of the training set for each allocation { n g } g ∈G .
5.1 Data externalities and dataset size (book rating prediction)
We ﬁrst examine how data externalities manifest across different possible sample sizes and ratios between groups, using the rating prediction task described above.
To compare performance across [Figure] many subsets of training data, we chose a model that is fast to train.
Following [7, 19], we use a linear regression model with (cid:96) 1 penalty (lasso), trained on 1000-dimensional tf-idf vector embeddings of the review texts.
The (cid:96) 1 penalty is chosen via a cross-validation search for each subset.
Figure 2 shows that negative data externalities can arise in real-data settings , evidenced by the decrease in AUROC evaluated one group as data from another group is added to the training set (left to right on horizontal axis).
As discussed in claim 1, these externalities provide obvious modeling interventions to increase per-group performance.
For each panel (evaluation group) in ﬁg. 2, we could subset the full training data to the allocation that maximizes AUROC along the vertical axis, resulting in two models trained with different training subsets, and as a result obtaining greater performance on each group (compared to performance with the full training set, n = 400 , 000 ).
The magnitude of the externality (measured from peak of each curve to rightmost point) is small, as expected from previous work that assumes between-group trends have a negligible effect on per-group performance as a function of training set allocations [7, 19].
Nonetheless, the existence of data externalities suggests that in some contexts, more nuanced scaling laws would be appropriate for describing model performance across data allocations.
The curves in ﬁg. 2 are not all monotonic, meaning there is not an “all or nothing” answer to whether merging or splitting training data from multiple source groups optimizes model performance: sometimes, the best performance for group A is achieved by adding a moderate amount of training samples from group B.
In fact, negative data externalities tend to manifest only once a certain number of points from the group in question are present in the training set (lighter-colored curves).
Drawing on our understanding from §4, we hypothesize that as the total number of training points increases, the training data “saturates” the model class, in the sense that the variance reduction due to additional points from group B is not worth the bias away from the optimal model parameters with respect to group A’s data distribution.
The exact saturation point would depend on the distribution of features and labels in each group and the distance between the group-optimal model parameters, the latter depending on the capacity of the model class.
To examine this further, our next set of experiments examines how data externalities manifest with models of different capacity.
5.2 Data externalities and model size (toxicity classiﬁcation)
We now examine how data externalities can differ for models of different sizes, using the CivilComments with identities dataset described above.
We ﬁne-tune pre-trained miniature BERT models of different model capacity from [21].
Model architectures are determined by the number of transformer layers L and hidden embeddings size H with corresponding number of attention heads.
For each
[Figure]
m a l
[Figure]
A U R O
[Figure]
m a x
i m u m
A U
R O C d e c r e a s e c o m p a r e d w i t
h u s i n g a ll d a t a
[Figure]
ﬁne-tuning run we use the Adam optimizer with learning rate 0.0001 and weight decay of 0.01 and train for 100 epochs with batchsize 64 and 500 gradient steps per epoch (see §A.2). The leftmost points in each top panel of ﬁg. 3 start with the maximum number of training points from the given group and increase the number of training points by adding data at random from the rest of the training set. The different hues in ﬁg. 3 rank the models in terms of number of overall parameters (see table 1 in [21]).
Negative slopes in the top row of ﬁg. 3 (and corresponding negative values in the bottom row of ﬁg. 3) evidence that data externalities arise in this data context and prediction task.
While increasing width or depth of the miniature BERT models (generally moving left to right on the bottom panels)
can decrease the exhibited data externalities on per-group AUROC, increasing model complexity does not necessarily mitigate negative data externalities on group performance, and in some cases can exacerbate them .
Adding additional layers to the model can increase the magnitude of data externalities evidenced (ﬁrst three bars of groups female, Muslim in ﬁg. 3, bottom), even though models with more layers tend to have higher overall performance.
While this phenomenon tends to be more stark for models of increasing depth, we caution interpretation of the relative merits of adding model capacity with either depth or width without further analysis.
Taken together, the results in §5.1 and §5.2 evidence that data externalities manifest in different real-world data contexts and shed light on when and why they might manifest.
Results in ﬁg.suggest that data externalities arise for one group primarily when there is enough representation in the training data from that group to “saturate” the model.
Results in ﬁg. 3 highlight that this point of saturation may depend on the complexity of the model parameterization among other factors.
Future work could leverage these ﬁndings to build a stronger understanding of how model capacity might be tailored to jointly increase performance while promoting data efﬁciency under different conditions.
Discussion and open questions
We have shown that data externalities, a phenomenon in which adding more data from some input sources reduces performance on key evaluation groups, can occur in many machine learning settings.
While this speciﬁc type of “data inefﬁciency” indicates room for model improvement, eq.
(4) is likely to be a coarse lower bound for the possible improvements that could be made.
Furthermore, the simple model modiﬁcation described in the proof of claim 1 is only computationally reasonable when the number of evaluation groups is relatively small.
Characterizing when and how data externalities can be (i) reliably identiﬁed for unknown evaluation groups or large number of groups and (ii) effectively mitigated within reasonable computational limits will be important future work.
We have focused on understanding how and when data externalities manifest across learning settings and training procedures.
It would be interesting interesting to study data externalities and dataefﬁciency more generally as a principle by which to design algorithms from the outset.
For example, a learning procedure guaranteeing no data externalities could enhance transparency regarding how input data affects model outputs, toward aligning of the goals of data minimization, participatory approaches and fairness with traditional performance optimization in machine learning
Acknowledgements
The authors thank Jilin Chen and Pranjal Awasthi for providing feedback during the conceptualization, development, and writing of this work.
We thank Yoni Halpern for feedback during the writing of this manuscript and Fernanda Viegas for feedback during ﬁgure development.
A Appendix A.1 Background and context for datasets
Goodreads.
The Goodreads datasets contain a corpus of book reviews across several genres, with numerical ratings and information about the book and reviewer [22].
This dataset and task have been used to study group effects across author genders and book genres in previous work [7, 19].
We follow a preprocessing approach similar to that in [19], where groups are genres.
Speciﬁcally, we instantiate our experimental dataset with reviews from two genres: history/biography ( history ) and fantasy/paranormal ( fantasy ), excluding the few books with overlap between these genres.
We take 250,000 review instances at random from the most popular 1000 books per genre, and split this data into group-balanced training set of 400,000 instances and validation/test set of 100,000 instances.
We consider binary labels of whether the rating accompanying each review text is a 5 star or less rating (originally measured on a 1-5 scale).
The class distribution similar between groups, with an average label of 0.39 across history group training instances, and 0.41 across fantasy training instances.
Review texts are embedded using tf-idf vectorization with 1000 features, and ignoring ‘english’ stopwords.
The tf-idf vectors are computed with respect to the entire 500,000 instances.
CivilComments (with identities) dataset for toxicity classiﬁcation in text.
Classifying toxic comments in a corpus of text can be challenging due to different meanings or sentiments of words or phrases when they are used in reference to certain groups or topics [9].
The CivilComments dataset with identity information [5] contains online comments with human-annotated labels corresponding to whether the comment is considered toxic and whether the comment is targeted at a speciﬁc group.
As noted in [17], the test set for some identity groups can be very small.
We focus on the four largest identity groups present in the dataset: female, male, Christian, and Muslim.
Groups are determined as binary labels if the annotator average is at least 0 .
5 for that identity group; binary toxicity labels are assigned similarly.
The training sets contain 405,130 total instances, with 53, instances corresponding to the female group, 44,848 instances corresponding to the male group, 40,423 instances corresponding to the Christian group, and 21,006 instances corresponding to the Muslim group.
The average toxicity rate (average binary label) differs across groups: in the training set the toxicity rates by group are: 14% (female), 15% (male), 11% (Christian), and 24% (Muslim).
To increase the number of evaluation samples across groups, we combine the validation and test sets in our analysis.
Combining the validation and test sets means we don’t have a separate validation set to tune model hyper-parameters.
Should a separate validation set be necessary, one could use the pre-processed dataset from [17], which combines some groups together to increase per-group sample sizes and allocates a larger fraction of the overall data to separate validation and test sets.
A.2 Additional experiment details A.2.1 Per-group AUROC
Evaluating AUROC within each group measures the strength of the relative ordering of the predictions within that group.
This is especially desirable for our experimental purposes should label distributions differ between groups, and is appropriate since we assume groups are known during training and evaluation.
We note the limitations of using per-group AUROC in other cases, when assigning per-group classiﬁcation thresholds is infeasible [5], e.g. when groups are assumed to be unknown
. A.2.2 Details on §5.
The pretrained miniature Bert models we use in §5.2 were accessed via https://tfhub.dev/ google/collections/bert/1 .
Details on the number of parameters, training time, and pretraining process for each model can be found in [21].
For each ﬁne-tuning run we use the Adam optimizer with learning rate 0.0001 and weight decay of 0.01 and train for 100 epochs with batchsize 64 and 500 gradient steps per epoch.
While we ﬁx these parameters as part of the ﬁxed training procedure deﬁnition for this experiment (e.g. as relates to deﬁnition 1), future work could incorporate hyper-parameter optimizations as part of the training procedures for each (dataset, model) pair.
A.3 Illustrative example details
Our examples in §4 use the following linear model:
[Equation]
Here we give the exact parameters that produce experimental results in ﬁg. 1.
In the ﬁrst example (ﬁg. 1a), we vary the intercept of the true model between the groups, as well as mean of the distribution of the features:
[Equation]
In the second example (ﬁg. 1b), the same model applies to both groups, but we vary the parameters governing observation noise (cid:15) g between groups, as well as the spread of the feature distribution:
[Equation]
B , (cid:15) B ∼ N (0 , 10) .