Are fairness metric scores enough to assess discrimination biases in machine learning? (2023)

Abstract: This paper presents novel experiments shedding light on the shortcomings of current metrics for assessing biases of gender discrimination made by machine learning algorithms on textual data. We focus on the Bios dataset, and our learning task is to predict the occupation of individuals, based on their biography. Such prediction tasks are common in commercial Natural Language Processing (NLP) applications such as automatic job recommendations. We address an important limitation of theoretical discussions dealing with group-wise fairness metrics: they focus on large datasets, although the norm in many industrial NLP applications is to use small to reasonably large linguistic datasets for which the main practical constraint is to get a good prediction accuracy. We then question how reliable are different popular measures of bias when the size of the training set is simply sufficient to learn reasonably accurate predictions. Our experiments sample the Bios dataset and learn more than 200 models on different sample sizes. This allows us to statistically study our results and to confirm that common gender bias indices provide diverging and sometimes unreliable results when applied to relatively small training and test samples. This highlights the crucial importance of variance calculations for providing sound results in this field.

## introduction potential biases introduced by artificial intelligence (ai) systems are now both an academic concern, but also a critical problem for industry, as countries plan to regulate ai systems that could adversely affect individual users. the so-called ai act foot_0 will require ai systems sold in the european union to have good statistical properties with regard to any potential discrimination they could engender. in particular, under the ai act, ai systems that exploit linguistic data like those for reviewing job candidates from text-based candidacies fall into the category of tightly regulated ai systems, as they are intended to be used for the recruitment or selection of natural persons (see annex iii of the ai act). such ai systems will require frequent and rigorous statistical testing for unwanted biases. 2 these regulatory advances have made it a pressing issue to define which metrics are appropriate for evaluating whether machine learning models can be considered fair algorithms in various industrial settings. in this context, we believe that these articles open at least two issues: (1) each fairness metric quantifies the fairness of a model in a different way and not all metrics are compatible with each other, as already discussed in @xcite @xcite . it is therefore easy to optimize its algorithm according to a single metric to claim fairness while overlooking all the other aspects of fairness measured by other metrics. (2) given that contemporary, theoretical discussions of fairness focus on large datasets but that the norm in many industrial 2 such ai systems are considered high-risk. the ai act (article 9.7) states: "the testing of the high-risk ai systems shall be performed, as appropriate, at any point in time throughout the development process, and, in any event, prior to the placing on the market or the putting into service. testing shall be made against preliminarily defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk ai system". article 10.2 specifies that "training, validation, and testing data sets shall be subject to appropriate data governance and management practices. those practices shall concern in particular, examination in view of possible biases" (among others). article 71 states that "non-compliance of the ai system with the requirements laid down in article 10 ... shall be subject to administrative fines of up to 30 000 000 eur or, if the offender is a company, up to 6 % of its total worldwide annual turnover for the preceding financial year". nlp applications is to use small linguistic datasets (ezen-can, 2020), one can wonder how reliable different popular measures of bias when the size of the training and validation sets is simply sufficient to learn reasonably accurate predictions. in general, this leads us to pose two questions, which are central to this paper: are fairness metrics always reliable on small samples, which are common in industrial contexts? how do they behave when applying standard debiasing techniques? to answer these questions, we propose a new experimental protocol to expose gender biases in nlp strategies, using variously sized subsamples of the bios dataset @xcite . we create 50 samples for each sample size (10k, 20k, 50k, and 120k) and train a model on each of the 200 samples. this gives us a mean and a variance on our results for all sample sizes to be able to compare them from a statistical point of view. we study the biases in these samples using three metrics; each sheds light on specific properties of gender bias. our study shows how bias is related to the training set size on a standard nlp dataset by revealing three points: first, commonly accepted bias indices appear unreliable when computed on ml models trained on relatively small training sets. moreover, our experiments reveal that the group parity gender gap metric (3.5) appears to be more reliable than other metrics on small samples. second, in the tested standard and large training sets, results are not homogeneous across professions and across the measures: sometimes there is gender bias against males, and sometimes against females in professions where one would expect something different. finally, the most traditional de-biasing methods, which consist in removing gender-sensitive words or replacing them with neutral variants, makes different metrics yield surprising and sometimes seemingly incompatible bias effects. we explain this phenomenon by the definitions of the metrics. in light of these findings, we think that one should use the main fairness metrics jointly to look for biases in smaller datasets and run enough models to have a variance. such bootstrapping procedures appear essential to robustly analyze how fair a prediction model is. our paper is structured as follows. section 2 surveys related work. section 3 introduces our experimental setup. section 4 discusses our results, with conclusions coming in section 5. section 6 discusses some of the limitations of our work. ## related work gender bias is pervasive in nlp applications: in machine translation @xcite @xcite , in hate speech detection @xcite , sentiment analysis @xcite , and in coreference resolution @xcite . gender bias with respect to classification has already been examined in @xcite @xcite @xcite , and reduced in @xcite @xcite . in particular, for the bert model, @xcite investigated gender bias. more generally, @xcite has studied the impact of increasingly large language models and has highlighted the sexist or racist biases and prejudices that result from them. however, the above-mentioned works only focused on single, large datasets. recently, a growing literature has started to propose to leverage statistical properties of fairness metrics, thus providing both sophisticated analysis and practically useful algorithms @xcite . in particular, a more rigorous statistical approach for bert models was introduced in @xcite . in this paper, we investigate the pertinence of different fairness metrics on samplings of different sizes out of a large dataset. we apply our principled statistical procedure and we present the results of these measures, along with their standard deviation and properties coming from student's t-tests. in addition to our scientific contribution, we have paid particular attention to the clarity of our explanations and the simplicity of our proposed protocol to allow small players to easily employ them for their real-world use cases. finally, our results attest to the importance of applying techniques of statistical analysis to fairness problems, and we hope that the guarantees gained through them provide a convincing argument for its more generalized application in the field. ## experimental protocol in this section, we detail the various components of our experimental setup. section 3.1 describes the dataset and section 3.2 the general type of model used to train the 200 models. section 3.3 introduces our debiasing technique used to illustrate our protocol. section 3.4 explains the sampling procedure and gives guarantees on the representativeness of the samples. finally, section 3.5 describes the different fairness metrics that we will compare and we justify these choices. ## results and discussion as shown in figure 1 , all the models we trained reached a prediction accuracy ranging from 0.72 to 0.86, as in (de-arteaga et al., 2019), which we consider as good since the classification problem involved 28 different occupations. all comparisons in this part were considered as significant by using student's t-tests (p-value of 0.05). we created two datasets without gender information: one version with all female gender indicators and the other with all male gender indicators. gender, therefore, has no impact on the finetuning part of our model. however, since we are starting from a pre-trained distilbert model (without a gender-neutral dataset), we had to check that the pre-training had no impact on the prediction. we therefore also made a student's test between the predictions of one model trained on the dataset with all the female gender indicators, and of another trained on the dataset with all the male gender indicators. the difference was not statistically significant; using one model or the other makes no difference. the analysis of the results of our protocol is made in two steps: a specific part and a general part. below in section 4.1, we analyze biases on two specific occupations, surgeon and physician. these two occupations are socially very interesting and their male/female distribution is very different, which is something we wanted to study. in section 4.2, we also observe the biases across the gamut of occupations in bios. all the results found in this preliminary study remain valid in a generalized case where we look at all the classes of the model. dividing our study like this allows us to discuss various details which support our key message without weighing down the article in the specific part while guaranteeing that our analysis is global and applies to the other classes of the model in the general part. ## conclusion our paper used the bios dataset to study the influence of the training set size on discriminatory biases. our results shed light on new phenomena: (1) fairness metrics did not converge to stable results for small sample sizes, which precluded any conclusions about the nature of the biases; (2) even on large training samples, the biases discovered were not always those expected and varied according to the metrics for several occupations; (3) a simple debiasing method, which consists in removing explicit gender indicators, had an unstable impact in our results depending on the metrics, though our analysis of the metrics can explain the instability. these results give two clear messages to data scientists who must design nlp applications with a potential social impact. they should first be particularly careful, as the decision rules they train may have unexpected discriminatory biases. in addition, a bias metric not only returns a score but has a strong practical meaning and may be unreliable, in particular when working with small training sets. so multiple metrics should be considered and statistical methods to obtain the variance of the observed metrics are necessary to guarantee the fairness of a model.