Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Abstract
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture : the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers.
While sharing offers advantages like amortizing effort, it also has risks.
We introduce and formalize one such risk, outcome homogenization : the extent to which particular individuals or groups experience the same outcomes across different deployments.
If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy.
We relate algorithmic monoculture and outcome homogenization by proposing the component sharing hypothesis : if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes.
We test this hypothesis on algorithmic fairness benchmarks, demonstrating that increased data-sharing reliably exacerbates homogenization and individual-level effects generally exceed group-level effects.
Further, given the current regime in AI of foundation models, i.e., pretrained models that can be adapted to myriad downstream tasks, we test whether model-sharing homogenizes outcomes across tasks.
We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization.
We also identify societal challenges that inhibit the measurement, diagnosis, and rectification of outcome homogenization in deployed machine learning systems.
Introduction
Machine learning is built on strong traditions of sharing: we share datasets (e.g. ImageNet), models (e.g. BERT), libraries (e.g. PyTorch), optimizers (e.g. Adam), evaluations (e.g. SuperGLUE) and much more.
This ethos of sharing serves the field well: we are able to repeatedly capitalize on the effort required to build high-quality assets (e.g. ImageNet has supported thousands of researchers in computer vision), and improvements to these assets have sweeping benefits (e.g. BERT raised all boats in NLP).
Yet does sharing also have risks?
Could this central tenet of the field lead to undesirable outcomes?
We observe that certain forms of sharing can be reinterpreted as monoculture: Kleinberg and Raghavan [2021] define algorithmic monoculture as the state "in which many decision-makers all rely on the [exact] same algorithm.
In parts of society where algorithmic systems are ubiquitous, we see trends towards such monoculture [Moore and Tambini, 2018, Engler, 2021].
Monocultures often pose serious risks: Kleinberg and Raghavan [2021] show monoculture is suboptimal for decision-makers when their decisions are interconnected, as when they compete to hire job candidates.
In ML, our sharing practices often are more complex than sharing the entire algorithmic system: should we think of our practices of sharing assets in ML as monoculture and, if so, what harms should we worry about?
We investigate this question by proposing one potential risk we call outcome homogenization , i.e. the phenomenon of individuals (or groups) exclusively receiving negative outcomes from all ML models they interact with.
For example, a job applicant may be rejected from every job they apply to due to the use of similar algorithmic resume screening systems at all companies.
We view outcome homogenization as an important class of systemic harms that arise when we study social systems , i.e. harms that require observing how individuals are treated by many decision-makers.
In §2, we conceptually motivate outcome homogenization in the context of algorithmic hiring.
In §3, we introduce the first mathematical formalism for outcome homogenization: we measure homogenization as the observed probability of systemic failure normalized by the base rate.
To link the practice of sharing in ML with the proposed harm of homogenization, we pose and test the component sharing hypothesis : algorithmic systems built using the same underlying components, such as training data and machine learning models, will tend to systematically fail for the same individuals or groups.
We see component sharing as a specific form of algorithmic monoculture, which broadens the initial definition in Kleinberg and Raghavan [2021] from decision-makers deploying the same system to deploying similar systems in terms of how they are constructed.
We investigate how two types of shared components — training data and foundation models — contribute to homogeneous outcomes.
In §4, we demonstrate that data-sharing reliably homogenizes outcomes for individuals and for racial groups, especially for small training datasets involving US census data.
In §5, we discuss how the rise of foundation models [Bommasani et al., 2021], i.e. pretrained models that can be adapted to myriad downstream tasks, could yield unprecedented homogenization.
Based on experiments with foundation models for vision (CLIP) and language (RoBERTa), to our surprise, we find the use of foundation models does not always exacerbate outcome homogenization.
Instead, we find the specific mechanism for adapting the foundation model to the downstream task significantly influences homogenization: for example, linear probing consistently leads to more homogeneous outcomes than finetuning for both modalities.
Through these experiments, it is clear that the relationship between sharing and homogenization is not fully explained by our hypothesis, but that there is some evidence that sharing homogenizes outcomes.
To advance the study of homogenization in practice, where systemic harms are most consequential, we conclude by identifying key challenges for diagnosing, measuring, and rectifying homogenization in society (§6).
Outcome Homogenization in Resume Screening
To illustrate outcome homogenization and its potential causes (including algorithmic monoculture), we will use the example of algorithmic resume screening.
Companies use resumes to screen job applicants, choosing which candidates to interview and which to reject.
Maximum homogenization occurs when every company makes the same decision about each candidate, such that each lucky candidate is interviewed by all companies and each unlucky candidate by no companies.
We say that the unlucky candidates who receive no interviews experience a systemic failure .
What factors might homogenize outcomes in human decision-making?
Even in the absence of the algorithms, we observe homogeneous outcomes in many setting.
In hiring, historically, hiring managers at each company decided who to interview and often agreed in their decisions.
This agreement can be attributed to multiple sources: first, if the needs of each company were identical, then managers at different companies may be incentivized to interview the same candidates, thereby homogenizing outcomes.
Second, if hiring managers’ choices are influenced by the same social biases, they will mistakenly reject the same people, thereby homogenizing their errors.
Bias in resume screening is well-documented and remains significant [Jowell and Prescott-Clarke, 1970, Bertrand and Mullainathan, 2004, Kline et al., 2021, inter alia ].
However, neither explanation implies that systemic failures are inevitable.
Since companies have different needs and resumes are imperfect predictors of success in role, the “best" candidates will likely differ across companies.
Further, bias is not uniform across companies: Kline et al.
[2021] find that 21% of firms were responsible for 46% of the racial bias in interview decisions.
Even if decisions are influenced by the same group-level biases, different companies may choose different individual members of the advantaged and disadvantaged groups.
Variance in company needs, in prevalence of bias, and in individual hiring manager preferences all make it more likely that different resumes survive the screening stage at different companies, ensuring some diversity in resume screening outcomes.
How do these dynamics change with the introduction of automated decision-making?
Most large companies now use automated resume screening software to parse resumes and decide which applicants advance.
As a stylized example, if every company deploys the same deterministic system and has the same hiring criteria, then outcomes will be necessarily homogeneous: individuals will either receive interviews at every company or be rejected by all of them (i.e. systemic failure).
This example is not far from reality: a few major vendors dominate the marketplace for algorithmic resume screening with 700 companies, including over 30% of Fortune 100 companies, relying on Hirevue [Hirevue, 2021].
Thiractice of different companies deploying the same system is defined as algorithmic monoculture by Kleinberg and Raghavan [2021].
More generally, different companies may instead deploy similar , but non-identical, systems.
We expand the definition of algorithmic monoculture to encapsulate this broader setting, which is also alluded to in Kleinberg and Raghavan [2021].
Engler [2021] describe this as the reality for college enrollment management algorithms, writing "there are a relatively small number (between five and 10) of prominent vendors in the enrollment management algorithm market, ...their process and analytics are markedly similar.
Since their processes seem relatively consistent, the outcomes might be as well — potentially leading to consistently good results for students who match the historical expectations of colleges, and consistently poor results for students who don’t".
Component Sharing Hypothesis.
In this work, we study systems that are related in how they are constructed, akin to what is described by Engler [2021].
We pose the component sharing hypothesis that relates such algorithmic monoculture with outcome homogenization: If deployed algorithmic systems share components, outcome homogenization will increase (i.e. there will be more systemic failures) .
In this work, we empirically test this hypothesis for two prominent forms of component sharing: (i) the sharing of training data in training all deployed systems (§4) and (ii) the sharing of the same foundation model for building all deployed systems (§5).
Formalizing Outcome Homogenization
While prior work [Kleinberg and Raghavan, 2021, Creel and Hellman, 2022] alludes to outcome homogenization, here we provide the first mathematical formalism of outcome homogenization.
In line with our running example of resume screening, we formalize outcome homogenization for individuals in terms of systemic failures (i.e. every algorithmic system fails for an individual).
We then generalize to the group setting, where groups are systemically excluded rather than individuals, with a discussion of how these metrics relate to established fairness, robustness, and accuracy metrics (§3.4).
3.1 Formalizing Outcome Homogenization for Individuals
Notation.
Since we define outcome homogenization as a systemic phenomenon, we consider a social system { h i } ki =1 where every individual j interacts each deployed models h i .
As an example, an indi- vidual j ∈ [ N ] submits features x ij (e.g. their resume) as input to company i ∈ [ k ] to receive an outcome
h
i ( x ij )=
ˆ y ij (e.g. an interview).
Let D i be the empirical distribution of inputs x i for company i .
To define a notion of failure, let I i ( x ij ) indicate if ˆ y ij is a negative outcome, i.e. individual j experiences a negative outcome from model
h i .
The failure rate for model h i is
[Equation]
Experimentally, we consider classification errors as failures (i.e. I
[Equation]
x
[Equation]
) ≜
[Equation]
h
[Equation]
( x [Equation] ) ̸ = y
[Equation]
(cid:3) ), but other negative outcomes (e.g. rejections from hiring or educational opportunities; I i ( x i ) ≜ I (cid:2)
h i ( x i )= − 1 (cid:3) ) also can be studied under our framework.
Systemic failures for individuals.
If an individual exclusively experiences failure, we say they experience systemic failure .
The observed rate of systemic failure SYSTEMIC FAILURE ( h 1 ,...,h k ) is
[Equation]
Homogenization metric for individuals.
SYSTEMIC FAILURE quantifies homogeneous outcomes, but is difficult to compare across systems with different underlying accuracies: SYSTEMIC FAILURE will in general be higher for less accurate systems independent of a specific tendency to pick (i.e. fail) on the same person.
While we may sometimes want to combine accuracy and outcome homogenization into an overall measure of utility or social welfare, which SYSTEMIC FAILURE ( h 1 ,...,h k ) implicitly does, we focus on a relative measure of homogenization that disentangles accuracy from homogenization.
In particular, we are interested in outcome homogenization even, and perhaps especially, in systems that are highly accurate.
As a result, we measure individual-level outcome homogenization for a social system (cid:8) h i (cid:9) k
i =1 by normalizing the observed rate of systemic failure by the expected rate of systemic failure.
[Equation]
This measure is the ratio between (i) the probability that an individual experiences systemic failure and (ii) the probability that randomly sampled outputs for each model are all failures.
That is, the measure captures how the rate of systemic failure changes when we attend to the structure of individuals.
3.2 Formalizing Outcome Homogenization for Groups
In addition to individual-level homogenization, we also measure group-level homogenization.
While our individual-level metric individualizes harm, complementing work on group-level biases, we may also want to identify the extent to which (possibly marginalized) social groups (e.g. Black women) are systemically excluded.
Further, we often lack individual-level information (e.g. due to privacy concerns; see §6), or study algorithmic deployments that do not share individuals (e.g. hiring in different states).
Notation.
For each input x i , denote the associated group as G ( x i ) ∈G .
Group identity can correspond to the data producer (e.g. the age of a user querying a search engine) or the data subject (e.g. the race of an individual subject to face recognition).
Let
D ig be the empirical distribution of inputs for group g (i.e. { x i | G ( x i )
= g } ).
The group failure rate FAIL g ( h i ) is
[Equation]
Homogenization metric for groups.
To measure group-level homogenization, we modify our individual-level metric with a weighted average over groups in place of a simple average over individuals. H group G
[Equation]
= (cid:88) g (cid:20)
[Equation]
i FAIL g
( h i ) (cid:21) (cid:81) i FAIL ( h i ) (5)
Weights.
We consider three weighting schemes, specified by categorical probability distributions
W distributed over G (full definitions in §A.1):
Average ( H avg ) W weights each group proportional to its frequency across all deployments.
Uniform ( H unif ) W is the uniform distribution, so W ( g )
= 1 |G| .
Worst ( H worst ) W assigns weight 1 to the group g worst with the highest systemic failure rate and 0 to all other groups.
This reduces the numerator to simply be the systemic failure rate for g worst .
We introduce these weight functions to clarify that, much like having both individual-level metrics and group-level metrics, we may want to weight groups differently in different circumstances.
For example, weighting by frequency may provide a useful overall measurement of homogenization but obscure systemic exclusion experienced by minority groups or specifically the worst-off group.
3.3 Understanding our metrics
As a ratio of probabilities, our metrics take values in [0 , ∞ ) where 0 indicates no systemic failures,indicates the observed rate matches the expected rate, and values greater than 1 indicate some degree of outcome homogenization.
In the individual setting, we assume each individual generates exactly one input per deployment, which may not hold in practice (e.g. people may submit multiple resumes or not apply to every company).
We appropriately generalize our individual-level metric to address this in Appendix A. Further, in the group setting, we recover the individual-level metric using the uniform weighting (or the average weighting) if each individual’s inputs are treated as belonging to their own group.
3.4 Relationship with other metrics
Since we introduce (several) metrics, we consider how they relate to metrics for related constructs (e.g. accuracy, fairness, robustness).
This speaks to the convergent and divergent validity of our metrics [Campbell and Fiske, 1959, Messick, 1987, Jacobs and Wallach, 2021], i.e. whether they are adequately correlated with metrics of similar constructs and adequately uncorrelated with metrics of dissimilar constructs.
Here, we discuss theoretical relationships, whereas in §5.2 we look at the empirical correlations.
Accuracy.
When failure are errors, we design our metrics to minimize (anti-)correlation with accuracy.
While not theoretically guaranteed, we empirically demonstrate this in Table 1.
With that said, we expect there will be settings where the two are correlated: our goal is not to technically ensure no correlation, but to ensure that we do not neglect homogeneous outcomes in highly accurate systems (i.e. neglect the individuals who are systemically failed even when the overall picture may seem rosy).
Fairness and Robustness.
Beyond accuracy, outcome homogenization is closely related to fairness and robustness.
However, we emphasize that outcome homogenization is fundamentally about correlated outcomes for social systems , whereas almost all robustness or fairness metrics are defined for a single model.
Recent work [Zhao and Chen, 2019, D’Amour et al., 2020, Wang et al., 2021] has initiated the study of fairness in multi-task learning, however these works focus on favorable overall trade-offs across tasks as opposed to systemic modes of failure.
Conversely, our metrics cease to be interesting (e.g. H individual is always 1 ) in the single-model setting as systemic failures are single-model failures.
At a more fine-grained level, algorithmic fairness metrics [e.g. Dwork et al., 2012, Hardt et al., 2016] emphasize discrepancies between individuals/groups .
In contrast, our metrics do not (explicitly) center these differences: we are interested in the observed rate of systemic failures (and whether this exceeds the expected rate).
Performance differences are not sufficient for outcome homogenization: if the performance disparities for each deployment do not align across deployments, then outcomes may not be homogeneous.
For robustness metrics, our metric H worst in the worst-case setting closely resembles the metrics studied in work on worst-group robustness [e.g. Sagawa* et al., 2020].
In particular, when there is only one deployment, our metric recovers the standard worst-group accuracy normalized by the overall accuracy.
3.5 Alternative metrics
In Appendix A, we more extensively discuss desiderata for our metric, alternatives we considered, and how we arrived at the metrics we present in the main paper.
With that said, we also note conditions where we may instead favor alternatives, as well as connections to familiar quantities like the covariance, Pearson correlation, and (pointwise) mutual information in the binary setting ( k =2 ).
Data-Sharing Experiments
Having stated our mathematical formalism and metrics for outcome homogenization, we test if sharing training data leads to outcome homogenization.
We consider widely used algorithmic fairness datasets [Fabris et al., 2022]: systemic failures for these specific datasets may not be of direct social consequence, but the datasets capture relevant social contexts where other forms of inequity have been documented.
Data.
We work with two datasets: German Credit [ GC ; Dua and Graff, 2017], the third most widely used fairness dataset, and ACS PUMS
[Ding et al., 2021], which was built to replace the most widely used fairness dataset, UCI Adult .
6 GC contains information on 1000 German contracts (e.g. credit history, credit amount, credit risk for the individual); following Wang et al.
[2021], we consider two prediction tasks of (i) predicting if the individual receives a good or bad loan and (ii) predicting whether their credit amount exceeds 2000.
ACS PUMS contains US Census survey data recordingfeatures (e.g. self-reported race and sex, occupation, average hours worked per week) for 3.6 million individuals.
Ding et al.
[2021] construct several predictions tasks of which we use three: (i) predict if an individual is employed, (ii) predict an individual’s income normalized by the poverty threshold, and (iii) predict if an individual has health insurance.
Individuals and Groups.
For both datasets, we have individual-level information, hence we measure individual-level homogenization across models for each task.
For ACS PUMS , we have self-identified race across 9 US Census categories (e.g. American Indian, Asian, Black/African American, White, two or more races), hence we measure group-level racial homogenization.
Experimental Design.
To test if data-sharing influences outcome homogenization, we execute a controlled comparison by specifying two sampling protocols for the training data: fixed and disjoint .
In general, articulating and determining what it precisely means for models to share data is complicated in natural settings: here we make the concept very precise.
In the fixed setting, we sample n points without replacement from the entire training dataset, which we use to train all of the k task-specific models ( k = for GC and k =3 for ACS PUMS ).
In the disjoint setting, we sample kn points without replacement that we randomly partition across the k task-specific models.
In other words, in the fixed setting, the task-specific models share the exact same training data inputs , whereas in the disjoint setting, the task-specific models share the same training distribution, but not the exact data .
We emphasize that this is a subtle difference between the setting, but it implies fixed shares more than disjoint .
Having specified the training data, we train models for each of the k tasks (we consider all of logistic regression, SVMs, gradient-boosted trees, and small neural networks in Appendix C).
To account for randomness, we report results averaged over 25 trials of the experiment (i.e. 5 samples of training data and 5 training runs per sample for every value of n we consider).
Results and Analysis.
In Figure 1 ( left ), we see clear evidence for our hypothesis: the fixed setting reliably shows more homogeneity than the disjoint setting.
Overall, across all 3 datasets and 4 learning algorithms we consider, we find that fixed generally leads to more homogeneous outcomes than disjoint , which provides evidence towards our hypothesis: the use of the same training data leads to greater outcome homogenization than the use of different (but identically distributed) training data.
However, this relationship is not perfect: we do see several instances where the degree of homogeneity is similar or even sometimes greater for the disjoint setting (e.g. regions of the left subplot of Figure 2).
Therefore, data sharing alone does not fully characterize homogeneity (e.g. randomness in training and instability in the number of observed systemic failures are important to consider).
Further, the trends in the error rates for fixed and disjoint are near-identical ( right subplot of Figure 1), as we would expect given the relationship between the sampling protocols.
That is, only observing the accuracy is not enough in these settings for understanding the observed systemic failures, and provides evidence that our measures correctly identify discrepancies even when the underlying accuracies of each model are (approximately) the same.
Finally, as a function of dataset size, we do not see consistent trends on the relationship between dataset scale and outcome homogenization, though we unsurprisingly see the disjoint and fixed settings converge for larger dataset sizes (since the discrepancies in the sampling become negligible due to convergence in measure).
Picking on the same person.
For ACS PUMS in Figure 2, we contrast the individual-level and grouplevel homogenization.
(Recall the average and uniform metrics are the group-level analogues of our individual-level metric.)
Outcomes are consistently more homogeneous at the individual-group than for racial groups.
In fact, group-level analysis show little homogenization (values near 1 ) and little change as a function of dataset size, whereas individual-level measurement exposes greater homogeneity and more variation (which is unsurprising since group-level quantities are more extensively aggregated).
This has significant ramifications for many works on algorithmic fairness, which only consider social groups (e.g. race): these works may miss systemic failures for particular individuals that are obscured at the group-level [cf.
Kearns et al., 2018, Hashimoto et al., 2018].
Even intersectional approaches may not sufficetosurfacethesesystemicfailures, unlesseach intersectionalgroupcomprisesasingleindividual.
Model-Sharing Experiments
Having found that data-sharing appears to exacerbate outcome homogenization, we now turn to model-sharing.
Specifically, we test how sharing foundation models affects outcome homogenization.
Bommasani et al.
[2021] define foundation models as "models trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks".
These models have had a sweeping impact on the AI research community, most notably in NLP, and are increasingly central to deploying ML at both startups (e.g. Hugging Face, Cohere, AI21) and established technology companies (e.g. Google, Microsoft, OpenAI).
Sharing is endemic to the foundation model paradigm: to justify their immense resource requirements, models must be used repeatedly for costs to amortize favorably.
In the extreme, if an entire domain like NLP comes to build almost all downstream systems on one or a few foundation models, then any biases or idiosyncrasies of these models that pervasively manifest downstream could potentially yield unprecedented systemic failures and outcome homogenization [Bommasani et al., 2021, Fishman and Hancox-Li, 2022].
We see initial evidence for such algorithmic monoculture: BERT was downloaded 10 million times in the past month 7 alone and GPTenables hundreds of deployed apps.
Consequently, we believe it is especially timely to understand if, and to what extent, outcomes get homogenized as these models become entrenched as infrastructure.
5.1 Experiments
Data.
To test how foundation models (e.g. CLIP, RoBERTa) influence homogenization, we run experiments for both vision and language data.
On the vision side, we work with the CelebA dataset [Liu et al., 2015] of celebrity faces paired with annotations for facial attributes.
For each face image, given the associated attributes, we define two tasks ( Earrings , Necklace ) that involve predicting whether the individual is wearing the specific apparel item.
Attribute prediction in CelebA has been studied previously in work on fairness and robustness [Sagawa* et al., 2020, Khani and Liang, 2021,
Wang et al., 2021].
On the language side, we use four standard English text classification datasets following Gururangan et al.
[2019]: IMDB
[Maas et al., 2011], AGNews [Zhang et al., 2015], Yahoo [Chang et al., 2008], and HateSpeech18 [de Gibert et al., 2018].
Individuals and Groups.
Since the vision tasks are all based on CelebA, we have individual-level information.
However, since the language tasks involve entirely different data (e.g. movie reviews vs. news articles), there is no (shared) individual-level information.
At the group-level, for vision we use annotations for hair color and for whether the individual has a beard , whereas for language we automatically group inputs by binary gender .
Experimental Design.
To test if model-sharing influences outcome homogenization, we contrast setting with differing degrees of model-sharing.
In the vision experiments, we produce task-specific models for each task by either (i) training from scratch on CelebA data, (ii) linearly probing by fitting a linear classifier on features from the CLIP foundation model [Radford et al., 2021], or (iii) finetuning CLIP.
To ensure meaningful comparisons, the models trained from scratch shared the same
ViT architecture [Dosovitskiy et al., 2021] used in CLIP but with weights initialized randomly.
In the language experiments, we further hone in on the specific adaptation method used to adapt the foundation model (specifically RoBERTa-base [Liu et al., 2019]) to each task.
We consider (i) linear probing , (ii) finetuning , and (iii) BitFiT
[Ben Zaken et al., 2022], which is a recent lightweight finetuning method in NLP that involves freezing all the RoBERTa weights except the bias parameters which are updated as in finetuning.
Consequently, BitFit is an intermediary between probing and finetuning, which has been shown to achieve similar accuracy as finetuning while updating very few of the pretrained parameters.
For both vision and for language, all models are trained for the same number of epochs and we repeat each experiment for 5 random seeds per adaptation method.
Hypotheses.
Much like data-sharing, model-sharing is graded and is not binary: different downstream systems can share varying degrees of underlying models.
By design, our experimental design suggests a continuum in sharing: first, downstream system either can share a foundation model or not ( scratch ).
Second, among methods that involve foundation models, all methods initialize the weights using the pretrained weights but differ in which parameters remain the same after adaptation is completed: finetuning changes all the parameters, BitFit only changes the bias parameters, and probing changes none of the parameters.
As a result, overall, we can rank methods from most to least sharing as (i) probing , (ii) BitFit , (iii) finetuning , (iv) scratch , which leads us to predict the degree of homogenization will also follow this ranking under our component-sharing hypothesis.
Results.
In Figure 3 ( left ), across all vision settings, we surprisingly find that scratch is the most homogeneous, i.e. more homogeneous than either approach involving shared foundation models.
This is the opposite of what we hypothesized: we posit that this may indicate model sharing is not the key explanatory variable for outcome homogenization here, but instead it is a more complex form of data sharing.
Specifically, we conjecture that since the scratch models are only trained on CelebA data, whereas the others also are trained on the much larger WebImageText via the CLIP foundation model, this may mean that the models based on CLIP are effectively regularized from learning idiosyncrasies of CelebA that the scratch models acquire.
This may more generally suggest that a more correct hypothesis around data sharing should factor in the relationship (e.g. distribution shift) between the training data and the evaluation data for each model.
Additionally, we find probing is consistently more homogeneous than finetuning , which aligns with our hypothesis.
Finally, akin to the census results (§4), we once again find that outcome homogenization is significantly higher for individuals than for groups (comparing to H avg and H unif ).
In Figure 3 ( right ), across all language settings, we find the ordering of homogenization matches what our hypothesis predicts.
Specifically, we find BitFit and finetuning achieve similar levels of homogeneity, even though BitFit updates 0 .
08% of the parameters full finetuning does (i.e. the number of shared parameters for BitFit is more like probing than finetuning), suggesting the number of shared parameters is not the right lens for understanding model sharing.
More broadly, these results do suggest parametersharing effects may contribute to outcome homogenization within the foundation model regime, but comparisons between foundation models and no foundation models may be more complex to explain.
5.2 Correlations between Metrics
Since we introduce several metrics, we measure the correlations between our metrics.
Further, we measure correlations with accuracy (specifically, the expected rate of systemic failure) to test if homogenization is disentangled from accuracy.
Since outcome homogenization is related to fairness, we also measure the correlation between our metrics and a standard group fairness metric.
Fairness metrics are generally defined for a single model h , whereas we study entire systems { h i } ki =1 .
We extend the unfairness definition used by Khani et al.
[2019] as the variance in the systemic failure rates across groups.
[Equation]
Results.
In Table 1, we report the pairwise correlation between metric pairs, based on the models
we
[Table]
trained in §5.1.
These correlations are for 45 systems (3 methods × 3 groupings × 5 random seeds) ofmodels for vision and 15 systems of 4 models for language.
For vision, our metrics are highly correlated with each other, whereas for language, H unif patterns quite differently (columns 1-3, 6-8).
This is to be expected in that the vision groups (e.g. hair colors)
all share similar frequencies, whereas the female group is significantly rarer in the language datasets.
For both language and vision, we find that our metrics are generally not correlated, or perhaps weakly correlated, with accuracy as we intended (columns 4, 9).
With respect to fairness, our worst-case metric H worst is strongly correlated for both modalities, but for the other two metrics we see no linear correlations and only monotone correlations for the vision experiments (columns 5, 10).
This is in line with our broader expectations that fairness and outcome homogenization are indeed related (especially for the worst-performing group), but that given they are distinct theoretical constructs, they should not always be correlated [Campbell and Fiske, 1959].
5.3 Discussion
Based on our findings for both data-sharing and model-sharing, we provide considerable evidence that sharing leads to homogeneous outcomes, but is insufficient as a full explanation.
That is, the simple notion of sharing needs refinement to fully characterize the observed homogeneity.
This is particularly relevant when the findings in §4 and §5 are contrasted, given model-sharing in the foundation model regime indirectlyimpliesimmensedata-sharingviathepretrainingdata(asmediatedbythepretrainedinitialization).Weemphasizethattheregimesforthesefindingsarequitedifferent:low-dimensionaltabulardata withsimplemodelfamiliesindata-sharingvs.high-dimensionalimages/textwithlargeneuralnetworks inmodel-sharing,sodiscrepanciesinthefindingsmaybeattributabletothesedifferences.
Morebroadly, webelieveamorecompleteexplanationrequiresaccountingforthedatadistributionsandtheassociated distributionshifts(e.g.betweenpretrainingandadaptation)atplay.
Whatwebelieveisclear,however,is thatourfindingsprovideanempiricalbasistobuildonourconceptualargumentsthatsharinginmachine learningcanincreasehomogenization.
Thismotivatesinvestigationintorealdeploymentsofmachine learning:forexample,doessharing/monocultureleadtohomogenizationinalgorithmichiring.
Societal Considerations and Challenges
To situate our work in a broader social context, we identify and discuss core challenges in diagnosing , measuring , and rectifying outcome homogenization.
Diagnosis.
In our work, we posit monoculture yields homogenization: to follow this approach would require knowing which deployments rely on the same vendor, dataset, or foundation model (i.e. knowing where there is monoculture).
Unfortunately, how algorithmic systems are constructed is often so opaque that identifying shared components is nigh impossible.
However, if high homogenization were demonstrated, the measurement itself could justify provisions for increased transparency to identify the latent monoculture (i.e. the anti-causal direction).
This provides a practical mechanism for empowering auditors to be granted conditional access to otherwise inaccessible proprietary systems.
Measurement.
Measuring homogenization only requires black box access, which is often achievable in practice [see Buolamwini and Gebru, 2018, Raji and Buolamwini, 2019, Metaxa et al., 2021].
However, identifying individual-level effects requires linking individual outcomes across deployments.
Due to privacy constraints, linking individuals across different deployments may be challenging or impossible, which motivates group-level homogenization as more generally accessible (see §3.2).
Rectification.
Even once outcome homogenization is identified, organizations may not be incentivized to reduce it.
When outcome homogenization does not align with each organization’s interests (e.g. maximizing accuracy), it may be necessary to introduce regulation, policy, or other compliance mechanisms to alter outcomes.
Navigating this trade-off is further complicated if the harms of homogeneous outcomes take time to observe or accrue, whereas the benefits of higher accuracy are immediate.
More optimistically, Kleinberg and Raghavan [2021] show under some conditions there is no trade-off between accuracy-maximizing policies and those that diversify outcomes for societal benefit.
Limitations and Conclusion
We have introduced, formalized, and measured outcome homogenization as a systemic harm that may arise from practices of sharing in ML.
Our work has an important limitation: our measures currently do not accommodate every setting and can be be brittle given systemic failures are themselves potentially rare events.
Furthermore, direct optimization of our measure may not lead to desirable outcomes, potentially contributing to ethics-washing.
The interpretation of our measure must be contextual : the implications and severity of homogeneous outcomes must be situated in a broader societal context.
Outcome homogenization is essential to comprehensively characterizing algorithmic harm at scale, especially given the trend towards algorithmic monoculture (e.g. via foundation models).
Without scrutiny, the harms of homogenization may become insidiously entrenched.
Consequently, we believe further study and early intervention is necessary to mitigate and prevent such harms in society.
Reproducibility.
All code, data, and experiments are available on Github and Codalab Worksheets.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Homogenization Metrics A.1 Group Homogenization Metrics
In §3.2, we introduced our group level homogenization metrics.
Specifically, we note the design decision of how to weight groups and the three weightings we consider: average , uniform , and worst .
Here, we provide the full mathematical definition of these metrics.
For convenience, let the frequency of group g in a specific dataset D i by denoted as p
i ( g ) , and the joint probability of the group across all datasets be denoted as p ( g ) ≜ (cid:81) ki
=1 p i ( g ) .
[Equation]
A.2 Relating Individual and Group Homogenization
In §4, we demonstrate empirically that outcomes can be more homogeneous for individuals than for racial groups.
Here we provide two scenarios that demonstrate circumstances where individual-level outcome homogenization is greater than, and is less than, group-level outcome homogenization.
(Of course, they can also be equal.)
In both settings, we will have two applications and two groups, where each group is comprised of two individuals.
As a result, H avg =
H unif , so we can compare either to
H individual .
In both settings, we will say Alice and Angelique are members of Group 1 and Bob and Bernardo are members of Group 2.
Scenario 1.
Let Alice and Bob be misclassified in Application 1 but not 2, and Angelique and Bernardo be misclassified in Application 2 but not 1.
No one is misclassified by both models, hence the number of observed systemic failures is 0 at the individual level, hence H individual = 0.
However, since there is a failure within Group 1 for both applications (and for Group 2 as well), the number of observed systemic failures is nonzero at the group level, hence H avg =
H unif > 0 .
Thus, in this scenario, we have seen that individual-level outcome homogenization can be less than group-level outcome homogenization.
Scenario 2.
Let Alice and Bob be misclassified in both applications, and Angelique and Bernardo be misclassified in neither application.
At the individual level, there are 2 systemic failures, so the observed rate of systemic failure is 24 =0 .
The overall error rate for each application is 0.5, so the expected rate of systemic failure is 0 .
5 × 0 .
5 = 0 .
Therefore, H individual = 0 .
25 = 2 .
At the group level, the observed rate of systemic failures is 0 .
25 for both groups.
The overall error rate for each application is still 0.5, so the expected rate of systemic failure is still 0 .
5 × 0 .
5 = 0 .
Therefore,
H avg =
H worst = 12 (0 .
+0 . 25) 0 .=1 .
Thus, in this scenario, we have seen that individual-level outcome homogenization can be greater than group-level outcome homogenization.
A.3 Generalizing Individual-Level Metric
In §3.3, we note that our individual-level framing assumes that every individual j produces inputs x ij for every company i .
The formalism in the main paper already permits these inputs to be different across companies for the same individual (e.g. Bob may submit different resumes when applying to
Microsoft and Google).
However, the formalism does not support two further general concepts: (i) multiple inputs per company and (ii) no inputs for some company (e.g. Bob does not apply to Amazon).
To accommodate the former, we note that the notion of failure can be modified depending on how the outcomes for the multiple inputs should be aggregated (e.g. a failure of a search engine may be determined by some fraction of search queries producing poor results for the user).
To address the latter concern, we introduce notation c j to indicate the subset of companies that individual j interacts with, i.e. c j ⊆ { 1 ,...,k } .
That is, any companies i ∈ { 1 ,...,k } that are not in c
j are those that individual j does not interact with.
Accordingly, we definite H individual as:
[Equation]
Notably, when ∀ j, c j =[ k ] , the denominator simplifies to (cid:81)
[Equation]
[ k ] FAIL ( h i ) , which matches Equation 3.
A.4 Alternative Metrics
In §3, we introduce the metrics we use to quantify outcome homogenization.
Of course, much like the many mathematical expressions that have been used to measure bias and fairness, there are many ways to reasonably measure homogenization.
Fundamentally, given the underlying construct of outcome homogenization is largely new, we begin by recognizing our understanding of the concept is incomplete and likely will require study in real systems to truly identify the precise desiderata for a measure.
In the interim, it is difficult to assess if the metric has structural fidelity [Loevinger, 1957], i.e. does the metric’s structure faithfully captures outcome homogenization?
Further, it is unclear if the metric has sufficient predictive validity to predict long-term outcomes (e.g. longitudinal harms arising from outcome homogenization) or how useful it is for testing specific scientific and social hypotheses [Jacobs and Wallach, 2021].
Ultimately, we believe the key test for the metric will be its consequential validity [Messick, 1987]: will the metric yield positive social impact as it "both reflects structure in the world and imposes structure upon the world"
[Hand, 2016].
To facilitate understanding, we transparently discuss other metrics we considered and why they may be preferable in some circumstances.
Ultimately, we worked with the metrics we describe in the paper as we found them to be the simplest and preferred their probabilistic interpretations, but we include reasons to prefer alternative as we describe them.
A.4.1 Alternative Metrics in the Binary Setting
Covariance and Pointwise Mutual Information.
When k = 2 , i.e. there are two companies, we note that our metric bears a very close resemblance to the covariance between the (indicator) random variables I
and I 2 .
In particular, the covariance is the difference of the quantities that define the ratio for our homogenization metric.
Similarly, our metric is the pointwise mutual information (PMI) evaluated at (1 , 1) up to the log .
[Equation] [Equation]
[Equation]
With respect to the covariance, we prefer that our metric is more naturally comparable across settings where the failure rates of social systems vary, whereas the covariance is more directly tied to the absolute scale of the failure rates.
With respect to the pointwise mutual information, we note that we are simply looking at the behavior of the social system in a special case where all models fail (which is one of the 2 k possible outcomes an individual could receive overall), whereas the overall
PMI considers all of them and is invariant to symmetries that are significant in our setting.
Further, both are traditionally studied in the binary setting, whereas we study behavior in settings where k> 2 .
Pearson Correlation.
Building on the relationship with the covariance, we note that our metric therefore also resembles the Pearson correlation .
[Equation]
In particular, when dealing with accurate models that are homogeneous (i
[Equation]
), the Pearson correlation coefficient approximates our metric up to the square root in the denominator.
Arguments can be made in favor and against this square root (and more generally a k -th root for k> 2 ); for simplicity we favor our metric that does not introduce such normalization but acknowledge this normalization may prove to be more favorable as the metric is further stress-tested.
A.4.2 Alternative Metrics beyond the Binary Setting
As we note in Footnote 2, our formalism and metrics are designed to be general, meaning that they can accommodate settings where the models h
i correspond to different tasks or scenarios.
(We make use of this generality in our experiments (§5) for vision and, especially, language.)
To permit this generality, we (reductively) binarize outcomes as either failures or not in our use of the indicator functions
I i .
In particular, for arbitrarily different tasks, the outcome spaces and their consequences on individuals may not be (easily) related.
In some settings, specifically those where the tasks that constitute the social system are sufficiently similar, we may instead prefer a more graded loss in the place of the binary notion of failures.
For each deployment by company i , denote the associated loss function as L i such that L i ( h i ( x ij ) ,y ij )=
ℓ ij is the loss experienced by individual j when interacting with model
h i .
In these settings, where the loss achieved across different applications is comparable, we can consider additional measures for homogenization in terms of this loss.
[Equation]
In words, the MinExp definition is the ratio of the average best-case loss for individuals with the loss of the best model h best and the ExpExp definition is the same but the denominator is the average loss of the models rather than the best loss.
These definitions bear close resemblance to the MaxMin and lexicographic (leximax and leximin) fairness definitions studied in the fairness literature [Dubins and Spanier, 1961, Henzinger et al., 2022]; the group-weighted analogue that use the worst group weighting recovers the MaxMin definition in the denominator.
(Note that the naming conventions are reversed since we define metrics in terms of loss whereas work in fairness and equitable allocations generally defines metrics in terms of utility.)
When the loss is the 0-1 classification loss, the MinExp definition and H individual are very similar: the numerators are the same (as the product of indicator variables is the same as their minimum) and the denominators are precisely z = (cid:81)
ki =1 FAIL ( h i ) FAIL ( h best ) factors of each other (i.e. the failure rate of all of the models except the best one).
Consequently, the MinExp yields values in the range [0 , 1] whereas
H individual is in [0 , ∞ ) .
Independent behavior across models in H individual is guaranteed to be 1 and maximal systemic failure in MinExp is guaranteed to be 1; symmetrically, independent behavior in
MinExp is z (which is non-constant) and maximal systemic failure in H individual is 1 z .
More broadly, the indicator variables we use in the main paper can be seen as a special case when using the 0-1 classification loss whereas arbitrary loss functions can be mapped to indicators by thresholding the loss (which does not require the loss to comparable, as different thresholds can be applied for different deployments).
As a result, both H individual and MinExp have clear merits; we believe MinExp may especially be a more natural definition where individuals have choices on which model h i to interact with of the k possible models.
In these settings (e.g. picking which voice assistant to use, or more generally consumer products), the losses will naturally be comparable and it may suffice for the individual to have one good option, which is more smoothly encoded by the minimum of the loss rather than the product of the indicators.
We encourage future work to explore whether the MinExp or H individual is preferable: in many settings we expect they will be strongly correlated given they are scalings of each other that depend not on the correlated nature of errors but the overall error rates, but they may be some settings where they diverge and one is clearly preferable to the other.
Currently, we recommend work to also consider MinExp when the losses are comparable, but to default to H individual since this is not required and H individual has a simple probabilistic interpretation.
B Reproducibility
All of the code required to train the models, group inputs for group-based metrics, measure homogenization, and generate visualizations will be released upon acceptance.
Additionally, to facilitate accessibility we will release a simple tool to compute our homogenization metrics.
We provide further experimental details below.
B.1 Census Experiments B.1.1 Data
We work with the ACS PUMS data introduced by Ding et al.
[2021], which contains US Census survey data for 3.6 million individuals.
Ding et al.
[2021] introduce the dataset to facilitate research into algorithmic fairness and the measurement of harms associated with algorithmic systems.
For each individual in the Census, 286 features are recorded (e.g. self-reported race and sex, occupation, average hours worked per week, marital status, healthcare status).
Ding et al.
[2021] construct several classification tasks using this data, where each task uses some of an individual’s features as inputs and one of their features as the label for the prediction task (e.g. predicting if an individual’s income exceeds $50000).
Of these tasks, we work with three in particular: ACSEmployment (predict if an individual is employed), ACSIncomePovertyRatio (predict an individual’s income normalized by the poverty threshold), and ACSHealthInsurance (predict if an individual has health insurance).
Following
Ding et al.
[2021], we split the data 80/20 for train/test.
We access this data through their folktables package.
10 We use the data for all of the US (they provided state-level data as well) from 2018, which is the primary data they analyze in their paper.
We select these tasks because Ding et al.
[2021] do not impose any filtering constraints in selecting data for these tasks, meaning all three tasks are posed for the same underlying individuals.
See Ding et al.
[2021] for more details.
License information is provided at https://github.com/zykls/folktables#license-and-terms-of-use and our use of the dataset adheres to these terms of service.
The data clearly can be personally identifying given it has census records for particular individuals, but there is no offensive content.
B.1.2 Models
Following Ding et al.
[2021], we train logistic regression models using sk-learn [Pedregosa et al., 2011] with default hyperparameters.
As a sanity check, we compared average model performance and saw it matched/exceeded what is reported in Ding et al.
For each setting ( fixed , disjoint ) and amount of training data, we trained 25 models across 5 random subsamples of training and 5 random seeds for model run per subsample, for each of the three tasks.
In aggregate, all of the models we trained took approximately 10 hours across 5 NVIDIA Titan Xp GPUs (or 50 hours on 1 NVIDIA Titan
Xp GPU), with additional experiments/debugging that is unreported in the paper taking approximately an additional 400 NVIDIA Titan Xp GPU hours.
B.1.3 Groupings
We consider racial groups, which are already provided in the dataset based on the self-identified category individuals chose in providing their information to the US Census.
The specific racial categories used are: White alone, Black/African American alone, American Indian alone, Alaska
Native alone, American Indian and Alaska Native, Asian alone, Native Hawaiian and Other Pacific
Islander alone, other unspecified race, two or more races.
B.2 Vision Experiments B.2.1 Data
We work with the CelebA dataset [Liu et al., 2015], which is a widely used dataset of celebrity faces paired with annotations for facial attributes.
For each face image, given the associated attributes, we define two tasks ( Earrings , Necklace ) that involve predicting whether the individual is wearing the specific apparel item.
Attribute prediction in CelebA has been studied previously in work on fairness and robustness [Sagawa* et al., 2020, Khani and Liang, 2021, Wang et al., 2021].
Given recent documentation of significant issues with computer vision datasets [e.g. Birhane and Prabhu, 2021, Birhane et al., 2021], we emphasize that we use the dataset solely for analytic reasons to study homogenization.
Further, given works like GenderShades [Buolamwini and Gebru, 2018] that highlight the harms of face recognition, we emphasize that we do not use the dataset for face recognition, but instead consider apparel prediction tasks where each apparel item/accessory is clearly observable in the face image.
In addition to Earrings and Necklace , the dataset also contains attributes for Eyeglasses and Neckties .
We initially included these tasks, but observed no individual was misclassified for all four tasks since these two tasks were very easy and models rarely produced any errors (e.g. the error rate for Eyeglasses was generally less than 1% ).
To be able to present non-trivial results for individual-level outcome homogenization, we therefore removed these tasks from consideration so that a nonzero number of systemic failures could be observed.
We downloaded the CelebA data from https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html .
We resized images to 224-by-224, and then apply the same augmentations as in CLIP [Radford et al., 2021], before feeding the image into the ViT-B/16 (which is what Radford et al.
[2021] do as well).
License information is provided here: https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html .
Our use of the dataset is consistent with the requirements for non-commercial research use.
The data clearly can be personally identifying given it has face images for particular individuals, but there is no offensive content.
B.2.2 Models
Following Radford et al.
[2021], we either use their released 150M parameter ViT-B/16 CLIP model (the largest publicly available CLIP model at the time of writing) or a randomly initialized model with the same architecture.
We used code from the official CLIP repository at https://github.com/opena i/CLIP .
Since Radford et al.
[2021] modify the standard Vision Transformer
[Dosovitskiy et al., 2021], we use their modified version for our scratch models.
As a sanity check of our implementation, we confirmed that our average finetuning accuracy were comparable to prior work (Sagawa* et al.
[2020] paper considersthetaskofpredictingBlondehair, withanImageNetpretrainedResNet-50, theyget94.8%and we get 95.9% in this particular task, i.e. when we also do the task of predicting Blonde hair with the same
ImageNet pretrained ResNet-50).
Further, on the Wearing Earrings task, we also confirmed that the
CLIP pretrained ViT-B/16 did better than a ResNet-50 (both ImageNet pretrained and CLIP pretrained).
For each setting ( scratch , probing , finetuning ), we trained 5 model runs with different seeds, for each of the tasks.
The final learning rates we used were 0.003 (training from scratch), 0.01 (linear probing), 0.000003 (fine-tuning), and they were selected by grid searching the learning rates on the Earrings task and we sanity checked that choosing a higher or lower learning rate led to lower accuracy.
We also train eachapproachfor10epochstoensuresimilarcomputationalresourcesareprovidedtoeachapproach.
In aggregate, all of the models we trained took approximately 1000 hours on one NVIDIA Titan Xp GPU.
B.2.3 Groupings
We consider groups based on hair color.
The dataset provides five hair-related annotations of Black,
Brown, Blonde, Grey, and Bald.
Since the Bald category was quite small, we collapsed the category with all examples that lacked a hair annotation (e.g. the hair color is obscured due to a hat) into an "Other" category to yield five total categories.
In addition, we measure outcome homogenization for individuals based on whether they have a beard .
While the CelebA dataset also contains annotations for race and gender, we chose to not look at these groups given we were concerned that the gender/race was being inferred by an annotator (crowdworker) from the face rather than being self-identified [Liu et al., 2015].
B.3 Language Experiments B.3.1 Data
We use the IMDB , AGNews , Yahoo , and HateSpeech18 datasets.
For IMDB , we were unable to find formal license information.
The data may contain some PII, but it is unlikely there is significant offensive content.
For AGNews , we were unable to find formal license information but found information indicating it should be used non-commercially, which we adhere to see.
The data may contain some PII, but it is unlikely there is significant offensive content.
For Yahoo , we were unable to find formal license information.
The data may contain some PII, especially given its nature, but it is unlikely there is significant offensive content.
For HateSpeech18 , we adhere to the license provided here: https://github.com/Vicomtech/hate-speech-dataset#license .
The data likely contains some PII given it is from forums, and certainly contains offensive content.
We access this data through Hugging Face Datasets [Lhoest et al., 2021].
The associated papers describe how the data was collected or scraped.
We tokenize the data using the RoBERTa
[Liu et al., 2019] tokenizer provided in Hugging Face Transformers
[Wolf et al., 2020].
B.3.2 Models
For all models we produced, we adapt RoBERTa-base [Liu et al., 2019] using the weights provided through Hugging Face Transformers [Wolf et al., 2020].
For all models, we use the default hyperparameters in the Trainer provide in the Transformer library, with the only change being a fairly standard setting of the learning rate to 2e-5.
As a sanity check of our implementation, we confirmed that our accuracy matches those provided in standard scripts/tutorials provided in Transformers and are quite similar to other works that work with these standard datasets [e.g. Gururangan et al., 2019].
For each setting ( probing , finetuning , BitFit ), we trained 5 model runs with different seeds, for each of the four tasks.
In aggregate, all of the models we discuss in the paper took approximately 36 hours across 5 NVIDIA
Titan Xp GPUs (or 180 hours on 1 NVIDIA Titan Xp GPU), with additional experiments/debugging that is unreported in the paper taking approximately an additional 2000 NVIDIA Titan Xp GPU hours.
B.3.3 Groupings
Since we consider four deployments that are largely unrelated to each other, there are no annotations of individuals or groups available that apply across all four datasets.
Consequently, we group inputs by (binary) gender, as this grouping applies across the four datasets.
Specifically, for each input we identify whether the input contains more references to the female gender (e.g. uses of words like "she"), the male gender, or no reference to an explicitly gendered term is made.
We acknowledge that this treats gender as a binary as part of an unfortunate trend in
NLP of works involving gender using binaries [Cao and
Daumé
III, 2020].
We use the peer-reviewed list of gender terms from
Garg et al.
[2018] and in accordance with the recommendations of
Antoniak and
Mimno [2021].
In the event that the same number of male and female gender terms are mentioned (possibly zero for both) in an input, we grouped the input in a third "Other" category.
While we did not extensively test, we did observe that the findings were not sensitive to small perturbations (i.e. random deletions of words from each list) of the lists we used.
Following
Antoniak and
Mimno [2021], we provide the exact lists below.
Male words =
{"he", "son", "his", "him", "father", "man", "boy", "himself", "male", "brother", "sons", "fathers", "men", "boys", "males", "brothers", "uncle", "uncles", "nephew", "nephews"}
Female words =
{"she", "daughter", "hers", "her", "mother", "woman", "girl", "herself", "female", "sister", "daughters", "mothers", "women", "girls", "femen", "sisters", "aunt", "aunts", "niece", "nieces"}
C Additional Experiments
Summary.
In §4, we report results on the ACS PUMS dataset.
Here, we supplement those findings to clarify whether the findings generalize across model families and to other datasets.
Qualitatively, across these additional evaluations, we do find the findings transfer: (i) the fixed partition of the data, where there is strictly greater data-sharing, reliably yields greater homogenization and (ii) when group-level data is available, individual-level homogenization exceeds group-level homogenization in magnitude.
Datasets.
In §4, we report results on the ACS PUMS dataset.
Here, we replicate the experiments performed in that section, but vary the dataset to clarify if the qualitative trends generalize to other datasets.
Recall that the structure of the data we deal with is fairly unusual: we are interested in datasets where each input is associated with multiple outcomes (i.e. the traditional multi-task learning setting) as we will share the training data across the models for each task.
However, because of our interests in social outcomes, we would further like each input to be meaningfully associated with a person (e.g. arbitrary multi-task learning datasets could be used but are unideal if they don’t additionally have this human-centric structure).
To identify additional relevant datasets, we survey datasets for multi-task learning, datasets for fairness in ML (which are generally human-centric as desired), and work at the intersection of fairness and multi-task ML.
Of these, we looked at Zhang and Yang [2017] for multi-task learning as well as https://paperswithcode.com/task/multi-task-learning , which provided a list ofdatasets.
We also looked at the 15 fairness datasets surveyed by Le Quy et al.
Finally, Wang et al.
[2021] initiated the study of multi-task fairness, considering four datasets.
From all of these, we arrived at four datasets with the structure we wanted: CelebA , UCI Adult , LSAC [Wightman et al., 1998], and GC [German Contracts; Dua and Graff, 2017].
All of these datasets have the desired structure: each input x ij is directly associated with an individual j and corresponds to multiple outcomes y 1 j ,...,y kj .
Of them, we report results for CelebA in §5.1, and we use the ACS PUMS dataset of Ding et al.
[2021] that was explicitly designed to supersede the similar US Census-based
UCI Adult .
Hence, we turn our attention to GC and LSAC .
The GC dataset contains information on 1000 German contracts, including credit history, credit amount, and the corresponding credit risk for that individual [Dua and Graff, 2017].
Following Wang et al.
[2021], the two prediction tasks we consider are (i) predicting if the individual receives a good or bad loan and (ii) predicting whether their credit amount exceeds 2000.
15 We additionally featurize in the same way as Wang et al.
[2021], using 16 attributes as features.
The data is accessed through the
UCI Machine Learning Repository
and we use an 80/20 train-test split like Wang et al.
All other experimental conditions match what we describe for the data-sharing experiments in §4 with further details given in §B.1.
The LSAC dataset was generated by the Law School Admission Council in the United States [Wightman et al., 1998].
This dataset contains information on 21,790 law students such as their entrance exam scores (LSAT) and their undergrad grade-point average (GPA) collected prior to law school.
From this, the two prediction tasks we consider are predicting (i) whether they pass the bar exam and (ii) whether their law school GPA exceeds the mean, directly following Wang et al.
The data is accessed through the tempeh 18 package with the default train-test split.
All other experimental conditions match what we describe for the data-sharing experiments in §4 with further details given in §B.1.
Model Families.
In §4, we report results using logistic regression as the model family.
Here, we replicate the experiments performed in that section, but vary the model family to clarify the influence of the model family in the homogenization of outcomes.
Specifically, we consider three additional model families: gradient boosted decision tree classifiers (GBM), 19 support vector machines, and neural networks.
All of these models are implemented using sk-learn [Pedregosa et al., 2011] with default parameters.
Results and Analysis.
We report additional results for ACS PUMS in Figures 4–6, for LSAC in Figures 7–9, and for GC in Figures 10–1.
Across these figures, we first establish that our core findings are upheld: (i) homogenization is reliably greater when there is more homogenized ( fixed ) than less ( disjoint ) across datasets and model families and (ii) homogenization is reliably greater when contrasting individual-level measures with the appropriate racial group-level measures.
However, we do note the absolute scale of homogenization is quite different across the datasets: we note that this is not surprising given the datasets are quite different, as are the relationship between the prediction tasks within a dataset.
Therefore, we highlight that our hypotheses make predictions about relative change (i e. sharing increases homogenization) but the underlying homogenization and absolute quantities will also depend significantly on the structure of the data and relationship between prediction tasks.
Further, for the LSAC and GC datasets, we visualize how the expected systemic failure rate (i e. the product of the error rates of the models, which is the denominator in our homogenization metric) changes as a function of data scale.
What we find is for both datasets, even one we have sufficient samples that the systemic failure rates in both the disjoint and fixed settings are identical (i e. variance due to finite sample effects in error rates becomes minimal), we see higher homogenization.
This demonstrates an important point: as the data grows, the expected systemic failure rate converges (as expected) to the same value for the disjoint and fixed partitions.
That is, if one pays attention only to the accuracies for each task, these systems are the same.
But even at this state, we see sizable discrepancies in outcome homogenization (i e. the observed systemic failure rates remain different) with the fixed partition displaying greater homogenization.
This drives home the point that data-sharing here has no effect on the accuracies of the resultant models, but that it does yield greater homogenization.
This demonstration is reminscent of work that demonstrates 21 the existence of models of (near) equal accuracy but that are simpler [Semenova et al., 2022] or more fair [Marx et al., 2020].
Here we observe a Rashomon effect [Breiman, 2001] at the level of social systems: both systems achieve the same accuracies, but one ( disjoint ) is more homogeneous than the other ( fixed ).