Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Anonymous Author(s)
AffiliationAddress email
Abstract
As the scope of machine learning broadens, we observe a recurring theme ofalgorithmic monoculture : the same systems, or systems that share components(e.g., datasets, models), are deployed by multiple decision-makers.
While sharingoffers advantages like amortizing effort, it also has risks.
We introduce and formalizeone such risk, outcome homogenization : the extent to which particular individualsor groups experience the same outcomes across different deployments.
If thesame individuals or groups exclusively experience undesirable outcomes, thismay institutionalize systemic exclusion and reinscribe social hierarchy.
We relatealgorithmic monoculture and outcome homogenization by proposing the componentsharing hypothesis : if algorithmic systems are increasingly built on the same data ormodels, then they will increasingly homogenize outcomes.
We test this hypothesison algorithmic fairness benchmarks based on the US Census, demonstrating thatincreased data-sharing exacerbates homogenization, especially for small datasets.
Further, given the current regime in AI of foundation models, i.e., pretrained modelsthat can be adapted to myriad downstream tasks, we test whether model-sharinghomogenizes outcomes across tasks.
We observe mixed results: we find that for bothvision and language settings, the specific methods for adapting a foundation modelsignificantly influence the degree of outcome homogenization.
We also identifysocietal challenges that inhibit the measurement, diagnosis, and rectification ofoutcome homogenization in deployed machine learning systems.
Introduction
Machine learning is built on strong traditions of sharing: we share datasets (e g., ImageNet), models(e.g., BERT), libraries (e.g., PyTorch), optimizers (e.g., Adam), evaluations (e.g., SuperGLUE) andmuch more.
This ethos of sharing serves the field well: we are able to repeatedly capitalize on the effortrequired to build high-quality assets (e g., ImageNet has supported thousands of researchers in computervision), and improvements to these assets have sweeping benefits (e g., BERT raised all boats in NLP).
Yet does sharing also have risks?
Could this central tenet of the field lead to undesirable outcomes
We forms sharing can be reinterpreted serious risks and is for when they compete to hire job candidates
Should we think of our practices of sharing assets in ML as monoculture and, if so what harms should we worry about
We investigate this question by proposing one potential risk we call outcome homogenization , i.e., thephenomenon of individuals (or groups) exclusively receiving negative outcomes from all ML modelsthey interact with.
We view outcome homogenization as an important class of systemic harms thatarise when we study social systems , i.e., harms that require observing how individuals are treated bymany decision-makers.
In §2, we conceptually motivate outcome homogenization in the context ofalgorithmic hiring.
In §3, we introduce the first mathematical formalism for outcome homogenization:we measure homogenization as the observed probability of systemic failure normalized by the base rate.
To link the practice of sharing in ML with the proposed harm of homogenization, we pose and testthe component sharing hypothesis : algorithmic systems built using the same underlying components,such as training data and machine learning models, will systematically exclude the same individualsor groups.
We see component sharing as a specific form of algorithmic monoculture, which generalizesthe definition in Kleinberg and Raghavan [2021] from decision-makers deploying the same systemto deploying similar systems in terms of how they are constructed.
We investigate how two types ofshared components — training data and foundation models — contribute to homogeneous outcomes.
In §4, we demonstrate that data-sharing reliably homogenizes outcomes for individuals and for racialgroups, especially for small training datasets involving US census data.
In §5, we discuss how the riseof foundation models [Bommasani et al., 2021], i.e., pretrained models that can be adapted to myriaddownstream tasks, could yield unprecedented homogenization.
Based on experiments with foundationmodels for vision (CLIP) and language (RoBERTa), to our surprise, we find the use of foundationmodels does not always exacerbate outcome homogenization.
Instead, we find the specific mechanismfor adapting the foundation model to the downstream task significantly influences homogenization:for example, linear probing consistently leads to more homogeneous outcomes than finetuning forboth modalities.
Through these experiments, it is clear that the relationship between sharing andhomogenization is not fully explained by our hypothesis, but that there is some evidence that sharinghomogenizes outcomes.
To advance the study of homogenization in practice, where systemic harmsare most consequential, we conclude by identifying key challenges for diagnosing, measuring, andrectifying homogenization in society (§6).
Outcome Homogenization in Resume Screening
To illustrate outcome homogenization and its potential causes, we will use the example of algorithmicresume screening.
Companies use resumes to screen job applicants, choosing which candidates tointerview and which to reject.
Maximum homogenization occurs when every company makes thesame decision about each candidate, such that each lucky candidate is interviewed by all companiesand each unlucky candidate by no companies.
We say that the unlucky candidates who receive nointerviews experience a systemic failure .
What factors might homogenize outcomes?
Historically, hiring managers at each company decidedwho to interview and often agreed in their decisions.
This agreement can be attributed to multiplesources: first, if the needs of each company were identical, then managers at different companies maybe incentivized to interview the same candidates, thereby homogenizing outcomes.
Second, if hiringmanagers’ choices are influenced by the same social biases, they will mistakenly reject the same people,thereby homogenizing their errors.
Bias in resume screening is well-documented and remains significant [Jowell and Prescott-Clarke, 1970, Bertrand and Mullainathan, 2004, Kline et al., 2021, inter alia ].
However, neither explanation implies that systemic failures are inevitable.
Since companies havedifferent needs and resumes are imperfect predictors of success in role, the “best" candidates will likelydiffer across companies.
Further, bias is not uniform across companies: Kline et al.
[2021] find that21% of firms were responsible for 46% of the racial contact gap.
Even if decisions are influencedby the same group-level biases, different companies may choose different individual members ofthe advantaged and disadvantaged groups.
Variance in company needs, in prevalence of bias, andin individual hiring manager preferences all make it more likely that different resumes survive thescreening stage at different companies, ensuring some diversity in resume screening outcomes.
How do these dynamics change with the introduction of automated decision-making?
Mostlarge companies now use automated resume screening software to parse resumes and decide whichapplicants advance.
As a stylized example, if every company deploys the same deterministic systemand has the same hiring criteria, then outcomes will be necessarily homogeneous: individuals willeither receive interviews at every company or be rejected by all of them (i.e., systemic failure).
This example is not far from reality: a few major vendors dominate the marketplace for algorithmicresume screening with 700 companies, including over 30% of Fortune 100 companies, relying on
Hirevue [Hirevue, 2021].
This practice of different companies deploying the same system is definedas algorithmic monoculture by Kleinberg and Raghavan [2021].
More generally, different companies may instead deploy similar , but non-identical, systems.
Weexpand the definition of algorithmic monoculture to encapsulate this broader setting, which is alsoalluded to in Kleinberg and Raghavan [2021].
Engler [2021] describe this as the reality for collegeenrollment management algorithms, writing "there are a relatively small number (between five and 10)of prominent vendors in the enrollment management algorithm market, ...their process and analyticsare markedly similar.
Since their processes seem relatively consistent, the outcomes might be as well— potentially leading to consistently good results for students who match the historical expectationsof colleges, and consistently poor results for students who don’t".
Component Sharing Hypothesis.
In this work, we study systems that are related in how they areconstructed, akin to what is described by Engler [2021].
We pose the component sharing hypothesisthat relates such algorithmic monoculture with outcome homogenization: If deployed algorithmicsystems share components, outcome homogenization will increase (i.e., there will be more systemicfailures) .
In this work, we empirically test this hypothesis for two prominent forms of componentsharing: (i) the sharing of training data in training all deployed systems (§4) and (ii) the sharing ofthe same foundation model for building all deployed systems (§5).
Formalizing Outcome Homogenization
While prior work [Kleinberg and Raghavan, 2021, Creel and Hellman, 2021, Bommasani et al., 2021]alludes to outcome homogenization, here we provide the first mathematical formalism of outcomehomogenization.
In line with our running example of resume screening, we formalize outcomehomogenization for individuals in terms of systemic failures (i.e., every algorithmic system fails foran individual).
We then generalize to the group setting, where groups are systemically excluded ratherthan individuals, with a discussion of how these metrics relate to established fairness, robustness, andaccuracy metrics (§3.4).
3.1 Formalizing Outcome Homogenization for Individuals
Notation.
Since we define outcome homogenization as a systemic phenomenon, we consider a socialsystem { h i } ki =1 where every individual j interacts with the k deployed models h 1 ,...,h k .
Specifically,an individual j ∈ [ N ] will submit features x ij (e.g., their resume) as input to company i ∈
[ k ] and receivean outcome h i ( x ij ) =
ˆ y ij (e.g., an interview).
Let D i be the empirical distribution of inputs x i forcompany i .
To define a notion of failure, let I i ( x ij ) indicate if ˆ y ij is a negative outcome, i.e., individual j experiencesa negative outcome from model
h i .
The failure rate for model h i is
[Equation]
Experimentally, we consider classification errors as failures (i.e., I i ( x i ) ≜
I (cid:2) h i ( x i )
̸
= y i (cid:3) ), but othernegative outcomes (e.g., rejections from hiring or educational opportunities; I i ( x i ) ≜ I (cid:2)
h i ( x i )= − 1 (cid:3) )also can be studied under our framework.
Systemic failures for individuals.
If an individual exclusively experiences failure, we say theyexperience systemic failure .
The observed rate of systemic failure S is
[Equation]
[2021] is related, but substantially distinct.
Concretely, their formalism considers harms experienced by decision-makers, whereas we center decision-subjects. Note framework is general: we permit the deployed models to be for different tasks and for the inputs to not be the
, though resume screening example all the models perform the same often submit the companies. In other words, it is the fraction of individuals who are failed by every model.
Homogenization metric for individuals.
S quantifies homogeneous outcomes but is difficultto compare across real-world systems with different underlying accuracies: S will in general behigher for less accurate systems independent of a specific tendency to pick on the same person.
While we may sometimes want to combine accuracy and outcome homogenization into an overallmeasure of utility or social welfare, which S implicitly does, we focus on a relative measure ofhomogenization that disentangles accuracy from homogenization.
In particular, we are interestedin outcome homogenization even, and perhaps especially, in systems that are highly accurate.
As a result, we measure individual-level outcome homogenization for a social system (cid:8) h i (cid:9) k
i =1 bynormalizing the observed rate of systemic failure by the expected rate of systemic failure.
[Equation]
This measure can be interpreted as the ratio between (i) the probability that an individual experiencessystemic failure and (ii) the probability that randomly sampled outputs for each model are allmisclassified.
That is, the measure captures how the rate of systemic failure changes when we attendto the structure of individuals.
3.2 Formalizing Outcome Homogenization for Groups
In addition to our individual-level metric, we also define a group-level metric for outcome homogenization.
While our individual-level metric individualizes harm, complementing work on group-levelharms like bias and inequity, we may also want to identify when broader social groups (e.g., Blackwomen) are systemically excluded.
This is especially relevant when we do not have access toindividual-level information (e.g., due to privacy concerns; see §6) or when individuals are not sharedacross data distributions (e.g., hiring in different states).
Notation.
For each input x i , we denote the associated group as G ( x i ) ∈ G .
Group identity maycorrespond with the data producer (e.g., the age of a user querying a search engine) or the datasubject (e.g., the race of an individual subject to face recognition).
Let D ig indicate restricting the datadistribution to inputs to group g ,
i.e., ∀ x i ∈ D ig ,G ( x i )=
g .
The group failure rate fail g ( h i ) is
[Equation]
Homogenization metric for groups.
To measure group-level outcome homogenization for a socialsystem (cid:8) h i (cid:9) k
i =1 , we modify our individual-level metric to compute a weighted average over groupsin place of a simple average over individuals.
[Equation]
Weight functions.
We consider three weight functions W : G → R (full definitions in §A.1):
Average ( H avg )
Each group g receives a weight proportional to its frequency across all deployments.
Uniform ( H unif )
Each group g receives equal weight.
Worst ( H worst )
The group with the highest rate of systemic failure receives a weight of 1 and all othergroups receive a weight of 0 .
We introduce these weight functions to clarify that, much like having both individual-level metricsand group-level metrics, we may want to weight groups differently in different circumstances.
Forexample, weighting by frequency may provide a useful overall measurement of homogenization butobscure systemic exclusion experienced by minority groups or specifically the worst-off group.
3.3 Understanding our metrics
As a ratio of probabilities, our metrics take values in [0 , ∞ ) where 0 indicates no systemic failures, 1indicates the observed rate matches the expected rate, and values greater than 1 indicate some degree ofoutcome homogenization.
In the individual setting, we assume each individual generates exactly oneinput per deployment, which may not hold in practice (e.g., people may submit multiple resumes or notapply to every company).
We appropriately generalize our individual-level metric to address this in Appendix A. Further, in the group setting, we recover the individual-level metric using the uniform weighting (or the average weighting) if each individual’s inputs are treated as belonging to their own group.
3.4 Relationship with related concepts
Given that we introduce (several) metrics, we want to be cognizant of how they relate to existingmetrics for related concepts (e.g., accuracy, fairness, robustness).
This speaks to the convergent anddivergent validity of our metrics [Campbell and Fiske, 1959, Messick, 1987, Jacobs and Wallach,2021], i.e., whether they are adequately correlated with metrics of similar constructs and adequatelyuncorrelated with metrics of dissimilar constructs.
Here, we discuss theoretical relationships, whereasin §5.2 we look at the empirical correlations.
Accuracy.
We design our metrics to minimize correlation with accuracy to separate homogenizationeffects explicable by the accuracy of the underlying models from those that go beyond what is predictedby accuracy.
While not theoretically guaranteed, we demonstrate in Table 1 that our metrics areconsistently uncorrelated, or very weakly correlated, with accuracy.
Fairness and Robustness.
Beyond accuracy, outcome homogenization is closely related to fairness androbustness.
However, we emphasize that outcome homogenization is fundamentally about correlatedoutcomes for social systems , whereas almost all robustness or fairness metrics are defined for a singlemodel.
Recent work [Zhao and Chen, 2019, D’Amour et al., 2020, Wang et al., 2021] has initiatedthe study of fairness in multi-task learning, however these works focus on favorable overall trade-offsacross tasks as opposed to systemic modes of failure.
Conversely, our metrics cease to be interesting(e.g.
, H individual is always 1 ) in the single-model setting as systemic failures are single-model failures.
At a more fine-grained level, algorithmic fairness metrics [e.g., Dwork et al., 2012, Hardt et al., 2016,
Corbett-Davies and Goel, 2018], as well as fairness metrics from other fields like the Gini coefficient,emphasize discrepancies between individuals/groups .
In contrast, our metrics do not (explicitly) centerthese differences: we are interested in the observed rate of systemic failures across all deployments(and whether this exceeds the expected rate).
Performance differences across individuals/groups arenot sufficient for outcome homogenization: if the performance disparities for each deployment do notalign across deployments, then the observed rate of systemic failure may not exceed the expected rate.
For robustness metrics, our metric H worst in the worst-case setting closely resembles the metrics studiedin work on worst-group robustness [e.g., Sagawa* et al., 2020].
In particular, when there is only onedeployment, our metric recovers the standard worst-group accuracy normalized by the overall accuracy.
3.5 Alternative metrics
In Appendix A, we more extensively discuss desiderata for our metric, alternatives we considered,and how we arrived at the metrics we present in the main paper.
With that said, we also note conditionswhere we may instead favor alternatives, as well as connections to familiar quantities like thecovariance, Pearson correlation, and (pointwise) mutual information in the binary setting ( k =2 ).
Data-Sharing Experiments
Having stated our mathematical formalism and metrics for outcome homogenization, we test if sharingtraining data leads to outcome homogenization.
We choose to look at the US Census, which has beenwell-studied in work on algorithmic fairness and ensures we have individual-level outcomes.
Data.
We work with the ACS PUMS data 3 introduced by Ding et al.
[2021], which contains US Censussurvey data recording 286 features (e.g., self-reported race and sex, occupation, average hours workedper week) for 3.6 million individuals.
Ding et al.
[2021] construct several classification tasks using this data, where each task uses some of an individual’s features as inputs and one of their features as thelabel for the prediction task (e.g., predicting if an individual’s income exceeds $50000).
We work withthree such tasks: ACSEmployment (predict if an individual is employed), ACSIncomePovertyRatio(predict an individual’s income normalized by the poverty threshold), and ACSHealthInsurance(predict if an individual has health insurance).
Individuals and Groups.
For each individual, we have their true label for each of the three tasks,enabling us to measure individual-level homogenization.
In the group setting, we measure howoutcomes homogenize when grouping by the 9 self-identified racial categories (e.g., American Indian,
Asian, Black/African American, White, two or more races).
Experimental Design.
To test if data-sharing influences outcome homogenization, we formulatea controlled comparison by specifying two settings for the training data: fixed and disjoint .
In thefixed setting, we sample n points without replacement from the entire ACS PUMS training datasetand set the training data for every task to this set.
In the disjoint setting, we sample 3 n points withoutreplacement and randomly partition them across the three tasks.
In other words, in the fixed setting,the three models are trained on data corresponding to exact same individuals, whereas in the disjointsetting, no pair of models is trained on data corresponding to the same individual.
Having specified thetraining data, we train logistic regression models for each of three tasks following Ding et al.
To account for randomness, we report results averaged over 25 trials of the experiment (i.e., 5 samplesof training data and 5 training runs per sample for every value of n we consider).
Results and Analysis.
In Figure 1, we present our results for this controlled comparison.
4 Acrossall of our homogenization metrics, we see clear trends (for both individuals and racial groups) that thefixed setting yields more homogeneous outcomes than the disjoint setting (i.e., blue line is above thecorresponding red line).
This provides clear evidence for our hypothesis: the use of the same trainingdata leads to greater outcome homogenization than the use of different (but identically distributed)training data.
Interestingly, our results also suggest that homogenization can be improved by randomlysubsampling a dataset.
Given a fixed dataset D , Figure 1 ( left ) suggests that homogenization is better(lower) if each of the three applications randomly subsamples a third of D rather than using the entiredataset D .
However, subsampling can reduce accuracy, which poses a tradeoff we discuss in §6.
Further, we see that these effects wane as the amount of training data increases (i.e., gap betweenblue and red lines from left to right).
This comes as no surprise: the two distributions converge dueto concentration of measure, hence the distinction between the training data in the fixed and in thedisjoint settings is increasingly small for larger amounts of data.
Picking on the same person.
Further, we contrast the degree of homogenization in the individualand group settings.
(Recall the average and uniform metrics are the group-level analogues of ourindividual-level metric.)
Outcomes are consistently more homogeneous at the individual-groupthan for racial groups.
This has significant ramifications for many works on algorithmic fairness,which only consider social groups (e.g., race): these works may miss systemic failures for particularindividuals that are obscured at the group-level [cf.
Kearns et al., 2018, Hashimoto et al., 2018].
Even intersectional approaches may not suffice to surface these systemic failures, unless each intersectionalgroup comprises a single individual.
Model-Sharing Experiments
Having found that data-sharing appears to exacerbate outcome homogenization, especially for smalldatasets, we now turn to model-sharing.
Specifically, we test how sharing pretrained models, orfoundation models [Bommasani et al., 2021], affects outcome homogenization.
Bommasani et al.[2021] define foundation models as "models trained on broad data (generally using self-supervisionat scale) that can be adapted to a wide range of downstream tasks".
These models have had a sweepingimpact on the AI research community, most notably in NLP, and are increasingly central to deploying
ML at both startups (e.g., Hugging Face, Anthropic, Inflection) and established technology companies(e.g., Google, Microsoft, Meta).
Sharing is endemic to foundation models: to justify their immenseresource requirements, the models must be used repeatedly for these costs to amortize favorably.
Inthe extreme, if an entire domain like NLP comes to build almost all downstream systems on one ora few foundation models, then any biases or idiosyncrasies of these models that pervasively manifestdownstream could potentially yield unprecedented systemic failures and outcome homogenization[Bommasani et al., 2021, Fishman and Hancox-Li, 2022].
We see initial evidence for such algorithmicmonoculture: BERT was downloaded 10 million times in the past month alone and GPT-3 enableshundreds of deployed apps.
Consequently, we believe it is especially timely to understand if, andto what extent, outcomes get homogenized as these models become entrenched as infrastructure.
5.1 Experiments
Data.
To test how foundation models influence homogenization, we run experiments for both visionand language data.
On the vision side, we work with the CelebA dataset [Liu et al., 2015] of celebrityfaces paired with annotations for facial attributes.
For each face image, given the associated attributes,we define two tasks ( Earrings , Necklace ) that involve predicting whether the individual is wearing thespecific apparel item.
Attribute prediction in CelebA has been studied previously in work on fairnessand robustness [Sagawa* et al., 2020, Khani and Liang, 2021, Wang et al., 2021].
On the language side,we use four standard English text classification datasets following Gururangan et al.
[2019]: IMDB[Maas et al., 2011], AGNews [Zhang et al., 2015], Yahoo [Chang et al., 2008], and HateSpeech18[de Gibert et al., 2018].
Individuals and Groups.
Since the vision tasks are all based on CelebA, we have individual-levelinformation.
However, since the language tasks involve entirely different data (e.g., movie reviewsvs.
news articles), there is no (shared) individual-level information.
At the group-level, for visionwe use annotations for hair color and for whether the individual has a beard , whereas for languagewe automatically group inputs by binary gender .
Experimental Design.
To test if model-sharing influences outcome homogenization, we contrastsetting with differing degrees of model-sharing.
In the vision experiments, we produce task-specificmodels for each task by either (i) training from scratch on CelebA data, (ii) linearly probing byfitting a linear classifier on frozen CLIP
[Radford et al., 2021] features, or (iii) finetuning CLIP.
Toensure meaningful comparisons, the models trained from scratch shared the same ViT architecture[Dosovitskiy et al., 2021] used in CLIP but with weights initialized randomly.
In the language experiments, we further hone in on the specific adaptation method used to adaptthe foundation model (specifically RoBERTa-base [Liu et al., 2019]) to each task.
We consider (i)linear probing , (ii) finetuning , and
(iii) BitFiT
[Ben Zaken et al., 2022], which is a recent lightweightfinetuning method in NLP that involves freezing all the RoBERTa weights except the bias parameterswhich are updated as in finetuning.
Consequently, BitFit is an intermediary between probing andfinetuning, which has been shown to achieve similar accuracy as finetuning while updating very fewof the pretrained parameters.
For both vision and for language, all models are trained for the samenumber of epochs and we repeat each experiment for 5 random seeds per adaptation method.
Hypotheses.
Much like data-sharing, model-sharing is graded and is not binary: different downstream systems can share varying degrees of underlying models.
By design, our experimental design
Vision: scratch is the most homogeneous, then probing , then finetuning .
Language: probing is the most homogeneous; finetuning and BitFit are similarly homogeneous.
suggests a continuum in sharing: first, downstream system either can share a foundation model ornot ( scratch ).
Second, among methods that involve foundation models, all methods initialize theweights using the pretrained weights but differ in which parameters remain the same after adaptationis completed: finetuning changes all the parameters, BitFit only changes the bias parameters, andprobing changes none of the parameters.
As a result, overall, we can rank methods from most to leastsharing as (i) probing , (ii) BitFit , (iii) finetuning , (iv) scratch , which leads us to predict the degreeof homogenization will also follow this ranking under our component-sharing hypothesis.
Results.
In Figure 2 ( left ), across all vision settings, we surprisingly find that scratch is the mosthomogeneous, i.e., more homogeneous than either approach involving shared foundation models.
Thisis the opposite of what we hypothesized: we posit that this may indicate model sharing is not the keyexplanatory variable for outcome homogenization here, but instead it is a more complex form of datasharing.
Specifically, we conjecture that since the scratch models are only trained on CelebA data,whereas the others also are trained on the much larger WebImageText via the CLIP foundation model,this may mean that the models based on CLIP are effectively regularized from learning idiosyncrasiesof CelebA that the scratch models acquire.
This may more generally suggest that a more correcthypothesis around data sharing should factor in the relationship (e.g., distribution shift) between thetraining data and the evaluation data for each model.
Additionally, we find probing is consistentlymore homogeneous than finetuning , which aligns with our hypothesis.
Finally, akin to the censusresults (§4), we once again find that outcome homogenization is significantly higher for individualsthan for groups (comparing to H avg and H unif ).
In Figure 2 ( right ), across all language settings, we find the ordering of homogenization matches whatour hypothesis predicts.
Specifically, we find BitFit and finetuning achieve similar levels of homogeneity, even though BitFit updates 0 .
08% of the parameters full finetuning does (i.e., the number of sharedparameters for BitFit is more like probing than finetuning), suggesting the number of shared parametersis not the right lens for understanding model sharing.
More broadly, these results do suggest parametersharing effects may contribute to outcome homogenization within the foundation model regime, butcomparisons between foundation models and no foundation models may be more complex to explain.
5.2 Correlations between Metrics
Since we introduce several metrics, we measure the correlations between our metrics.
Further,we measure correlations with accuracy (specifically, the expected rate of systemic failure) to test ifhomogenizationisdisentangledfromaccuracy.
Sinceoutcomehomogenizationisrelatedtofairness, wealso measure the correlation between our metrics and a standard group fairness metric.
Fairness metricsare generally defined for a single model h , whereas we study entire systems { h i } ki =1 .
We extend thefairness definition used by Khani et al.
[2019] as the variance in the systemic failure rates across groups.
[Equation]
Results.
In Table 1, we report the pairwise correlation between metric pairs, based on the models wetrained in §5.1.
These correlations are for 45 systems (3 methods × 3 groupings × 5 random seeds) of 2models for vision and 15 systems of 4 models for language.
For vision, our metrics are highly correlated [Table] with each other, whereas for language, H unif patterns quite differently (columns 1-3, 6-8).
This isto be expected in that the vision groups (e.g., hair colors) all share similar frequencies, whereas thefemale group is significantly rarer in the language datasets.
For both language and vision, we find thatour metrics are generally not correlated, or perhaps weakly correlated, with accuracy as we intended(columns 4, 9).
With respect to fairness, our worst-case metric H worst is strongly correlated for bothmodalities, but for the other two metrics we see no linear correlations and only monotone correlationsfor the vision experiments (columns 5, 10).
This is in line with our broader expectations that fairness andoutcome homogenization are indeed related (especially for the worst-performing group), but that giventhey are distinct theoretical constructs, they should not always be correlated [Campbell and Fiske, 1959]. 5.3 Discussion Based on our experimental findings for both data-sharing and model-sharing, we provide considerableevidence that sharing in various forms does contribute to homogenization.
However, we also provide nuancewithourfindings, indicatingthatthesimplenotionofsharingwillneedtoberefinedtoproperlycapture the full effects observed in terms of homogenization. This is particularly relevant when the findingsin §4 and §5 are contrasted with each other, given model-sharing in the foundation model regime alsoimpliesimmensedata-sharingintheformofthepretrainingdata(asmediatedbythepretrainedinitialization). We emphasize that the regimes we study are considerably different: low-dimensional tabular datawith simple model families in the data-sharing experiments vs. high-dimensional images/text with largeand deep neural networks in the model-sharing, so it is not surprising the results do not fully coincide. Most broadly, we believe a more complete explanation will require factoring in the nature of the variousdata distributions and the associated distribution shifts (e.g., between pretraining and adaptation) at play.
What we believe is clear, however, is that our findings provide an empirical basis to build on our conceptual arguments that sharing in ML can increase homogenization, and that this motivates exploration intowhether these effects manifest in deployed high-stakes systems like those used for algorithmic hiring.
To situate our work in a broader social context, we identify and discuss core challenges in diagnosing ,measuring , and rectifying outcome homogenization.
Diagnosis.
To diagnose homogenization as a byproduct of monoculture requires knowing whichcompanies rely on the same vendor, dataset, or foundation model.
How specific algorithmic systemsare constructed is often so opaque that determining the responsibility of a shared component foroutcome homogenization is nigh impossible.
However, if a high outcome homogenization score wasmeasured across deployments, the measurement itself could justify a request for increased transparencyand access.
We believe this could be a useful practical mechanism for balancing tensions in auditingto provide conditional access to auditors.
Measurement.
Measuring homogenization only requires black box access to algorithmic systems,which is often achievable in practice [see Buolamwini and Gebru, 2018, Raji and Buolamwini, 2019,
Metaxa et al., 2021].
However, identifying individual-level effects requires observing outcomes forthe same individual across deployments.
Due to privacy constraints, correlating individuals acrossdatasets from different companies might be challenging or impossible.
Such constraints incentivizethe study of homogenization for groups as a more accessible concept, as in §3.2.
Rectification.
Even once outcome homogenization is identified, organizations deploying systemsresponsible for homogeneous outcomes may not be incentivized to reduce it.
When outcomehomogenization does not align with each organization’s interests (e.g., maximizing accuracy), itmay be necessary to introduce regulation, policy, or other compliance mechanisms to encourage organizations to alter their outcomes.
Navigating this trade-off is further complicated if the harms ofhomogeneous outcomes take time to observe or accrue.
More optimistically, Kleinberg and Raghavan[2021] show that in some instances, there may not be a trade-off between policies optimal for eachorganization and those that diversify outcomes for societal benefit.
Limitations and Conclusion
We have introduced, formalized, and measured outcome homogenization as a systemic harm thatmay arise from practices of sharing in ML.
Our work has an important limitation: our measure cannotaccommodate every setting and may be brittle given that systemic failures are often rare.
Furthermore,direct optimization of our measure may not lead to desirable outcomes, potentially contributing toethics-washing.
The interpretation of our measure must be contextual : the implications and severityof homogeneous outcomes must be situated in a broader societal context.
Outcome homogenization is essential to comprehensively characterizing algorithmic harm atscale, especially given the trend towards algorithmic monoculture.
Without scrutiny, the harms ofhomogenization may become insidiously entrenched.
Consequently, we believe further study andearly intervention is necessary to mitigate and prevent such harms in society.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] ,or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)? [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes](b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A](d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
In §3.2, we introduced our group level homogenization metrics. Specifically, we note the designdecision of how to weight groups and the three weightings we consider: average , uniform , and worst . Here, we provide the full mathematical definition of these metrics.
For convenience, let the frequencyof group g in a specific dataset D i by denoted as p i ( g ) and the joint probability of the group acrossall datasets be denoted as p ( g ) ≜ (cid:81) ki =1 p i ( g ) .
[Equation]
In §4, we demonstrate empirically that outcomes can be more homogeneous for individuals than forracial groups. Here we provide two scenarios that demonstrate circumstances where individual-leveloutcome homogenization is greater than, and is less than, group-level outcome homogenization.
(Ofcourse, they can also be equal.) In both settings, we will have two applications and two groups, whereeach group is comprised of two individuals. As a result, H avg = H unif , so we can compare either to
H individual . In both settings, we will say Alice and Angelique are members of Group 1 and Bob and Bernardoare members of Group 2. Scenario 1. Let Alice and Bob be misclassified in Application 1 but not 2, and Angelique and Bernardobe misclassified in Application 2 but not 1.
No one is misclassified by both models, hence the numberof observed systemic failures is 0 at the individual level, hence H individual = 0. However, since there is afailure within Group 1 for both applications (and for Group 2 as well), the number of observed systemicfailures is nonzero at the group level, hence H avg = H unif > 0 . Thus, in this scenario, we have seenthat individual-level outcome homogenization can be less than group-level outcome homogenization.
Scenario 2. Let Alice and Bob be misclassified in both applications, and Angelique and Bernardobe misclassified in neither application.
At the individual level, there are 2 systemic failures, so theobserved rate of systemic failure is 2 4 =0 . The overall error rate for each application is 0.5, so theexpected rate of systemic failure is 0 . 5 × 0 . 5 = 0 .
Therefore, H individual = 0 . 25 = 2 . At the grouplevel, the observed rate of systemic failures is 0 . 25 for both groups.
The overall error rate for eachapplication is still 0.5, so the expected rate of systemic failure is still 0 . 5 × 0 . 5 = 0 . Therefore,
{+H avg = H worst = 12 (0 . 
 +0 . 25) 0 .=1 . Thus, in this scenario, we have seen that individual-level outcomehomogenization can be greater than group-level outcome homogenization.+}
In §3.3, we note that our individual-level framing assumes that every individual j produces inputsx ij for every company i . The formalism in the main paper already permits these inputs to be differentacross companies for the same individual (e.g., Bob may submit different resumes when applyingto Microsoft and Google). However, the formalism does not support two further general concepts:(i) multiple inputs per company and (ii) no inputs for some company (e.g., Bob does not apply to
Amazon).
To accommodate the former, we note that the notion of failure can be modified dependingon how the outcomes for the multiple inputs should be aggregated (e.g., a failure of a search enginemay be determined by some fraction of search queries producing poor results for the user). To address the latter concern, we introduce notation c j to indicate the subset of companies thatindividual j interacts with, i.e., c j ⊆{ 1 ,...,k } .
That is, any companies i ∈{ 1 ,...,k } that are not in c jare those that individual j does not interact with. Accordingly, we definite H individual as: [Equation] Notably, when ∀ j, c j =[ k ] , the denominator simplifies to (cid:81)
[Equation]
fail ( h i ) , which matches Equation 3. A.4 Alternative Metrics In §3, we introduce the metrics we use to quantify outcome homogenization. Of course, much like themany mathematical expressions that have been used to measure bias and fairness, there are many waysto reasonably measure homogenization.
Fundamentally, given the underlying construct of outcomehomogenization is largely new, we begin by recognizing our understanding of the concept is incompleteand likely will require study in real systems to truly identify the precise desiderata for a measure.
In the interim, it is difficult to assess if the metric has structural fidelity [Loevinger, 1957], i.e., doesthe metric’s structure faithfully captures outcome homogenization? Further, it is unclear if the metrichas sufficient predictive validity to predict long-term outcomes (e.g., longitudinal harms arising fromoutcome homogenization) or how useful it is for testing specific scientific and social hypotheses[Jacobs and Wallach, 2021]. Ultimately, we believe the key test for the metric will be its consequentialvalidity [Messick, 1987]: will the metric yield positive social impact as it "both reflects structure inthe world and imposes structure upon the world" [Hand, 2016].
To facilitate understanding, we transparently discuss other metrics we considered and why they maybe preferable in some circumstances.
Ultimately, we worked with the metrics we describe in the paperas we found them to be the simplest and preferred their probabilistic interpretations, but we includereasons to prefer alternative as we describe them.
A.4.1 Alternative Metrics in the Binary Setting
Covariance and Pointwise Mutual Information.
{+When k = 2 , i.e., there are two companies, wenote that our metric bears a very close resemblance to the covariance between the (indicator) randomvariables I 
+}
and I 2 .
In particular, the covariance is the difference of the quantities that define theratio for our homogenization metric.
Similarly, our metric is the pointwise mutual information (PMI)evaluated at (1 , 1) up to the log .
[Equation]
[Equation] [Equation] With respect to the covariance, we prefer that our metric is more naturally comparable across settingswhere the failure rates of social systems vary, whereas the covariance is more directly tied to theabsolute scale of the failure rates. With respect to the pointwise mutual information, we note thatwe are simply looking at the behavior of the social system in a special case where all models fail(which is one of the 2 k possible outcomes an individual could receive overall), whereas the overall
PMI considers all of them and is invariant to symmetries that are significant in our setting. Further,both are traditionally studied in the binary setting, whereas we study behavior in settings where k> 2 . Pearson Correlation. Building on the relationship with the covariance, we note that our metrictherefore also resembles the Pearson correlation .
[Equation]
In particular, when dealing with accurate models that are homogeneous (i.e., E (cid:2) I 1 I 2 (cid:3) >>
E (cid:2) I 1 (cid:3) E (cid:2) I 2 (cid:3) ), the Pearson correlation coefficient approximates our metric up to the square root inthe denominator. Arguments can be made in favor and against this square root (and more generallya k -th root for k> 2 ); for simplicity we favor our metric that does not introduce such normalization butacknowledge this normalization may prove to be more favorable as the metric is further stress-tested.
As we note in Footnote 2, our formalism and metrics are designed to be general, meaning that they canaccommodate settings where the models h i correspond to different tasks or scenarios. (We make use ofthis generality in our experiments (§5) for vision and, especially, language.)
To permit this generality,we (reductively) binarize outcomes as either failures or not in our use of the indicator functions
I i .
In particular, for arbitrarily different tasks, the outcome spaces and their consequences on individualsmay not be (easily) related.
In some settings, specifically those where the tasks that constitute the social system are sufficientlysimilar, we may instead prefer a more graded loss in the place of the binary notion of failures.
Foreach deployment by company i , denote the associated loss function as L i such that L i ( h i ( x ij ) ,y ij )
=
ℓ ijis the loss experienced by individual j when interacting with model
h i .
In these settings, where theloss achieved across different applications is comparable, we can consider additional measures forhomogenization in terms of this loss.
[Equation]
In words, the MinExp definition is the ratio of the average best-case loss for individuals with theloss of the best model h best and the ExpExp definition is the same but the denominator is the averageloss of the models rather than the best loss.
These definitions bear close resemblance to the MaxMinand lexicographic (leximax and leximin) fairness definitions studied in the fairness literature [Dubinsand Spanier, 1961, Henzinger et al., 2022]; the group-weighted analogue that use the worst groupweighting recovers the MaxMin definition in the denominator.
(Note that the naming conventionsare reversed since we define metrics in terms of loss whereas work in fairness and equitable allocationsgenerally defines metrics in terms of utility.)
When the loss is the 0-1 classification loss, the MinExp definition and H individual are very similar:the numerators are the same (as the product of indicator variables is the same as their minimum) andthe denominators are precisely z =
[Equation]
factors of each other (i.e., the failure rate of all ofthe models except the best one).
Consequently, the MinExp yields values in the range [0 , 1] whereas
H individual is in [0 , ∞ ) .
Independent behavior across models in H individual is guaranteed to be 1 andmaximal systemic failure in MinExp is guaranteed to be 1; symmetrically, independent behavior in
MinExp is z (which is non-constant) and maximal systemic failure in H individual is 1 z .
More broadly, the indicator variables we use in the main paper can be seen as a special case when usingthe 0-1 classification loss whereas arbitrary loss functions can be mapped to indicators by thresholdingthe loss (which does not require the loss to comparable, as different thresholds can be applied fordifferent deployments).
As a result, both H individual and MinExp have clear merits; we believe MinExpmay especially be a more natural definition where individuals have choices on which model h i tointeract with of the k possible models.
In these settings (e.g., picking which voice assistant to use,or more generally consumer products), the losses will naturally be comparable and it may suffice forthe individual to have one good option, which is more smoothly encoded by the minimum of the lossrather than the product of the indicators.
We encourage future work to explore whether the MinExp or H individual is preferable: in many settingswe expect they will be strongly correlated given they are scalings of each other that depend not on thecorrelated nature of errors but the overall error rates, but they may be some settings where they divergeand one is clearly preferable to the other.
Currently, we recommend work to also consider MinExpwhen the losses are comparable, but to default to H individual since this is not required and H individual hasa simple probabilistic interpretation.
B Reproducibility
All of the code required to train the models, group inputs for group-based metrics, measurehomogenization, and generate visualizations will be released upon acceptance.
Additionally, tofacilitate accessibility we will release a simple tool to compute our homogenization metrics.
Weprovide further experimental details below.
B.1 Census ExperimentsB.1.1 Data
We work with the ACS PUMS data introduced by Ding et al.
[2021], which contains US Censussurvey data for 3.6 million individuals.
Ding et al.
[2021] introduce the dataset to facilitate researchinto algorithmic fairness and the measurement of harms associated with algorithmic systems.
Foreach individual in the Census, 286 features are recorded (e.g., self-reported race and sex, occupation,average hours worked per week, marital status, healthcare status).
Ding et al.
[2021] construct severalclassification tasks using this data, where each task uses some of an individual’s features as inputs andone of their features as the label for the prediction task (e.g., predicting if an individual’s income exceeds$50000).
Of these tasks, we work with three in particular: ACSEmployment (predict if an individualis employed), ACSIncomePovertyRatio (predict an individual’s income normalized by the povertythreshold), and ACSHealthInsurance (predict if an individual has health insurance).
Following
Ding et al.
[2021], we split the data 80/20 for train/test.
We access this data through their folktablespackage.
7 We use the data for all of the US (they provided state-level data as well) from 2018, whichis the primary data they analyze in their paper.
We select these tasks because Ding et al.
[2021] donot impose any filtering constraints in selecting data for these tasks, meaning all three tasks are posedfor the same underlying individuals.
See Ding et al.
[2021] for more details.
License informationis provided at https://github.com/zykls/folktables#license-and-terms-of-use andour use of the dataset adheres to these terms of service.
The data clearly can be personally identifyinggiven it has census records for particular individuals, but there is no offensive content.
B.1.2 Models
Following Ding et al.
[2021], we train logistic regression models using sk-learn [Pedregosa et al.
,2011] with default hyperparameters.
As a sanity check, we compared average model performance andsaw it matched/exceeded what is reported in Ding et al.
For each setting ( fixed , disjoint ) andamount of training data, we trained 25 models across 5 random subsamples of training and 5 randomseeds for model run per subsample, for each of the three tasks.
In aggregate, all of the models we trained took approximately 10 hours across 5 NVIDIA Titan Xp GPUs (or 50 hours on 1 NVIDIA Titan
Xp GPU), with additional experiments/debugging that is unreported in the paper taking approximatelyan additional 400 NVIDIA Titan Xp GPU hours.
B.1.3 Groupings
We consider racial groups, which are already provided in the dataset based on the self-identifiedcategory individuals chose in providing their information to the US Census.
The specific racialcategories used are: White alone, Black/African American alone, American Indian alone, Alaska
Native alone, American Indian and Alaska Native, Asian alone, Native Hawaiian and Other Pacific
Islander alone, other unspecified race, two or more races.
B.2 Vision ExperimentsB.2.1 Data
We work with the CelebA dataset [Liu et al., 2015], which is a widely used dataset of celebrity facespaired with annotations for facial attributes.
For each face image, given the associated attributes,we define two tasks ( Earrings , Necklace ) that involve predicting whether the individual is wearingthe specific apparel item.
Attribute prediction in CelebA has been studied previously in work onfairness and robustness [Sagawa* et al., 2020, Khani and Liang, 2021, Wang et al., 2021].
Givenrecent documentation of significant issues with computer vision datasets [e.g., Birhane and Prabhu,2021, Birhane et al., 2021], we emphasize that we use the dataset solely for analytic reasons to studyhomogenization.
Further, given works like GenderShades [Buolamwini and Gebru, 2018] that highlightthe harms of face recognition, we emphasize that we do not use the dataset for face recognition, butinstead consider apparel prediction tasks where each apparel item/accessory is clearly observable in theface image.
In addition to Earrings and Necklace , the dataset also contains attributes for Eyeglassesand Neckties .
We initially included these tasks, but observed no individual was misclassified forall four tasks since these two tasks were very easy and models rarely produced any errors (e.g., theerror rate for Eyeglasses was generally less than 1% ).
To be able to present non-trivial results forindividual-level outcome homogenization, we therefore removed these tasks from consideration sothat a nonzero number of systemic failures could be observed.
We downloaded the CelebA data fromhttps://mmlab.ie.cuhk.edu.hk/projects/CelebA.html .
We resized images to 224-by-224,and then apply the same augmentations as in CLIP
[Radford et al., 2021], before feeding the image intothe ViT-B/16 (which is what Radford et al.
[2021] do as well).
License information is provided here:https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html .
Our use of the dataset is consistentwith the requirements for non-commercial research use.
The data clearly can be personally identifyinggiven it has face images for particular individuals, but there is no offensive content.
B.2.2 Models
Following Radford et al.
[2021], we either use their released 150M parameter ViT-B/16 CLIP model (thelargest publicly available CLIP model at the time of writing) or a randomly initialized model with thesame architecture.
We used code from the official CLIP repository at https://github.com/openai/CLIP .
Since Radford et al.
[2021] modify the standard Vision Transformer
[Dosovitskiy et al., 2021],we use their modified version for our scratch models.
As a sanity check of our implementation, we confirmed that our average finetuning accuracy were comparable to prior work (Sagawa* et al.
[2020] paperconsidersthetaskofpredictingBlondehair, withanImageNetpretrainedResNet-50, theyget94.8%andweget95.9% inthisparticulartask, i.e.,whenwealso dothetaskof predictingBlondehairwiththe same
ImageNet pretrained ResNet-50).
Further, on the Wearing Earrings task, we also confirmed that the
CLIP pretrained ViT-B/16 did better than a ResNet-50 (both ImageNet pretrained and CLIP pretrained).
For each setting ( scratch , probing , finetuning ), we trained 5 model runs with different seeds, for eachof the tasks.
The final learning rates we used were 0.003 (training from scratch), 0.01 (linear probing),0.000003 (fine-tuning), and they were selected by grid searching the learning rates on the Earrings taskand we sanity checked that choosing a higher or lower learning rate led to lower accuracy.
We also traineachapproachfor10epochstoensuresimilarcomputationalresourcesareprovidedtoeachapproach.
Inaggregate, all of the models we trained took approximately 1000 hours on one NVIDIA Titan Xp GPU.
B.2.3 Groupings
We consider groups based on hair color.
The dataset provides five hair-related annotations of Black,
Brown, Blonde, Grey, and Bald.
Since the Bald category was quite small, we collapsed the category withall examples that lacked a hair annotation (e.g., the hair color is obscured due to a hat) into an "Other"category to yield five total categories.
In addition, we measure outcome homogenization for individualsbased on whether they have a beard .
While the CelebA dataset also contains annotations for race andgender, we chose to not look at these groups given we were concerned that the gender/race was beinginferred by an annotator (crowdworker) from the face rather than being self-identified [Liu et al., 2015].
B.3 Language ExperimentsB.3.1 Data
We use the IMDB , AGNews , Yahoo , and HateSpeech18 datasets.
For IMDB , we were unable tofind formal license information.
The data may contain some PII, but it is unlikely there is significantoffensive content.
{+For AGNews , we were unable to find formal license information but foundinformation indicating it should be used non-commercially, which we adhere to see. 
+}
The data maycontain some PII, but it is unlikely there is significant offensive content.
For Yahoo , we were unableto find formal license information.
The data may contain some PII, especially given its nature, but itis unlikely there is significant offensive content.
For HateSpeech18 , we adhere to the license providedhere: https://github.com/Vicomtech/hate-speech-dataset#license .
The data likelycontains some PII given it is from forums, and certainly contains offensive content.
We access thisdata through Hugging Face Datasets [Lhoest et al., 2021].
The associated papers describe how thedata was collected or scraped.
We tokenize the data using the RoBERTa
[Liu et al., 2019] tokenizerprovided in Hugging Face Transformers [Wolf et al., 2020].
B.3.2 Models
For all models we produced, we adapt RoBERTa-base [Liu et al., 2019] using the weights providedthrough
Hugging Face Transformers [Wolf et al., 2020].
For all models, we use the default hyperparameters in the Trainer provide in the Transformer library, with the only change being a fairly standard settingof the learning rate to 2e-5.
As a sanity check of our implementation, we confirmed that our accuracymatches those provided in standard scripts/tutorials provided in Transformers and are quite similarto other works that work with these standard datasets [e.g., Gururangan et al., 2019].
For each setting( probing , finetuning , BitFit ), we trained 5 model runs with different seeds, for each of the four tasks.
In aggregate, all of the models we discuss in the paper took approximately 36 hours across 5 NVIDIA
Titan Xp GPUs (or 180 hours on 1 NVIDIA Titan Xp GPU), with additional experiments/debuggingthat is unreported in the paper taking approximately an additional 2000 NVIDIA Titan Xp GPU hours.
B.3.3 Groupings
Since we consider four deployments that are largely unrelated to each other, there are no annotations of individuals orgroups available that apply across all four datasets.
Consequently, we group inputs by (binary) gender, as this grouping applies across the four datasets.
Specifically, foreach input we identify whether the input contains more references to the female gender (e.g., uses of words like"she"), the male gender, or no reference to an explicitly gendered term is made.
We acknowledge that thistreats gender as a binary as part of an unfortunate trend in
NLP of works involving gender using binaries [Cao and
Daumé
III, 2020].
We use the peer-reviewed list of gender terms from
Garg et al.
[2018] and in accordance withthe recommendations of
Antoniak and
Mimno [2021].
Inthe event that the same number of male and female gender terms are mentioned (possibly zero for both) in an input, we grouped the input in a third "Other" category.
Whilewe did not extensively test, we did observe that the findings were not sensitive to small perturbations (i.e., random deletions of words from each list) of the lists weused.
Following
Antoniak and
Mimno [2021], we provide the exact lists below.947 Male words = {"he", "son", "his", "him", "father", "man", "boy", "himself", "male", "brother", "sons","fathers", "men", "boys", "males", "brothers", "uncle", "uncles", "nephew", "nephews"}
Female words =
{"she", "daughter", "hers", "her", "mother", "woman", "girl", "herself", "female", "sister", "daughters", "mothers", "women", "girls", "femen", "sisters", "aunt", "aunts", "niece", "nieces"}
C Additional Experiments
Summary.
In §4, we report results on the ACS PUMS dataset.
Here, we supplement those findings toclarify whether the findings generalize across model families and to other datasets.
Qualitatively, acrossthese additional evaluations, we do find the findings transfer: (i) the fixed partition of the data, wherethere is strictly greater data-sharing, reliably yields greater homogenization and (ii) when group-leveldata is available, individual-level homogenization exceeds group-level homogenization in magnitude.
Datasets.
In §4, we report results on the ACS PUMS dataset.
Here, we replicate the experimentsperformed in that section, but vary the dataset to clarify if the qualitative trends generalize to otherdatasets.
Recall that the structure of the data we deal with is fairly unusual: we are interested in datasetswhere each input is associated with multiple outcomes (i.e., the traditional multi-task learning
setting)as we will share the training data across the models for each task.
However, because of our interestsin social outcomes, we would further like each input to be meaningfully associated with a person(e.g., arbitrary multi-task learning datasets could be used but are unideal if they don’t additionallyhave this human-centric structure).
To identify additional relevant datasets, we survey datasets for multi-task learning, datasets for fairnessin ML (which are generally human-centric as desired), and work at the intersection of fairness andmulti-task ML.
Of these, we looked at Zhang and Yang [2017] for multi-task learning as well ashttps://paperswithcode.com/task/multi-task-learning , which provided a list of 51datasets.
We also looked at the 15 fairness datasets surveyed by Le Quy et al.
Finally, Wanget al.
[2021] initiated the study of multi-task fairness, considering four datasets.
From all of these, we arrived at four datasets with the structure we wanted: CelebA , UCI Adult , LSAC[Wightman et al., 1998], and GC [German Contracts; Dua and Graff, 2017].
All of these datasetshave the desired structure: each input x ij is directly associated with an individual j and corresponds tomultiple outcomes y 1 j ,...,y kj .
Of them, we report results for CelebA in §5.1, and we use the ACS PUMSdataset of Ding et al.
[2021] that was explicitly designed to supersede the similar US Census-based
UCI Adult .
Hence, we turn our attention to GC and LSAC .
The GC dataset contains information on 1000 German contracts, including credit history, creditamount, and the corresponding credit risk for that individual [Dua and Graff, 2017].
Following Wanget al.
[2021], the two prediction tasks we consider are (i) predicting if the individual receives a goodor bad loan and (ii) predicting whether their credit amount exceeds 2000.
11 We additionally featurizein the same way as Wang et al.
[2021], using 16 attributes as features.
The data is accessed through the
{+UCI Machine Learning Repository 
+}
and we use an 80/20 train-test split like Wang et al.
All other experimental conditions match what we describe for the data-sharing experiments in §4 withfurther details given in §B.1.
The LSAC dataset was generated by the Law School Admission Council in the United States [Wightman et al., 1998].
This dataset contains information on 21,790 law students such as their entrance examscores (LSAT) and their undergrad grade-point average (GPA) collected prior to law school.
From this,the two prediction tasks we consider are predicting (i) whether they pass the bar exam and (ii) whethertheir law school GPA exceeds the mean, directly following Wang et al.
The data is accessedthrough the tempeh 14 package with the default train-test split.
All other experimental conditionsmatch what we describe for the data-sharing experiments in §4 with further details given in §B.1.
Model Families.
In §4, we report results using logistic regression as the model family.
Here, we replicate the experiments performed in that section, but vary the model family to clarify the influence of themodel family in the homogenization of outcomes.
Specifically, we consider three additional model families: gradient boosted decision tree classifiers (GBM), 15 support vector machines, and neural networks.
All of these models are implemented using sk-learn [Pedregosa et al., 2011] with default parameters.
Results and Analysis.
We report additional results for ACS PUMS in Figures 3–5, for LSACin Figures 6–8, and for GC in Figures 9–12.
Across these figures, we first establish that our corefindings are upheld: (i) homogenization is reliably greater when there is more homogenized ( fixed )than less ( disjoint ) across datasets and model families and (ii) homogenization is reliably greater whencontrasting individual-level measures with the appropriate racial group-level measures.
However, wedo note the absolute scale of homogenization is quite different across the datasets: we note that this is not surprising given the datasets are quite different, as are the relationship between the prediction taskswithin a dataset.
Therefore, we highlight that our hypotheses make predictions about relative change(i e., sharing increases homogenization), but the underlying homogenization and absolute quantitieswill also depend significantly on the structure of the data and relationship between prediction tasks.
Further, for the LSAC and GC datasets, we visualize how the expected systemic failure rate (i e., theproduct of the error rates of the models, which is the denominator in our homogenization metric)changes as a function of data scale.
What we find is for both datasets, even one we have sufficientsamples that the systemic failure rates in both the disjoint and fixed settings are identical (i e., variance due to finite sample effects in error rates becomes minimal)
we see higher homogenization.
Thisdemonstrates an important point: as the data grows the expected systemic failure rate converges(as expected) to the same value for the disjoint and fixed partitions.
That is, if one pays attentiononly to the accuracies for each task, these systems are the same.
But even at this state, we seesizable discrepancies in outcome homogenization (i e., the observed systemic failure rates remaindifferent) with the fixed partition displaying greater homogenization.
This drives home the pointthat data-sharing here has no effect on the accuracies of the resultant models but that it does yield greater homogenization.
This demonstration is reminscent of work that demonstrates 17 the existenceof models of (near) equal accuracy but that are simpler [Semenova et al., 2022] or more fair [Marxet al., 2020].
Here we observe a Rashomon effect [Breiman, 2001] at the level of social systems: bothsystems achieve the same accuracies, but one ( disjoint ) is more homogeneous than the other ( fixed ).