Detecting intersectionality in NER models: A data-driven approach (2023)

Abstract: The presence of bias is a pressing concern for both engineers and users of language technology. What is less clear is how exactly bias can be measured, so as to rank models relative to the biases they display. Using an innovative experimental method involving data augmentation, we measure the effect of intersectional biases in Danish models used for Name Entity Recognition (NER). We quantify differences in representational biases, understood as a systematic difference in error or what is called error disparity. Our analysis includes both gender and ethnicity to illustrate the effect of multiple dimensions of bias, as well as experiments which look to move beyond a narrowly binary analysis of gender. We show that all contemporary Danish NER models perform systematically worse on non-binary and minority ethnic names, while not showing significant differences for typically Danish names. Our data augmentation technique can be applied on other languages to test for biases which might be relevant for researchers applying NER models to the study of cultural heritage data.

## introduction issues of bias and discrimination are essential in contemporary natural language processing (nlp). research has consistently pointed to bias in word embeddings @xcite , and for downstream tasks such as coreference resolution @xcite , and language generation @xcite . several survey papers have also mapped out the landscape of bias research in the field of nlp, showing a lack of clear definitions of bias and normative motivation in nlp bias research @xcite ; and further emphasising the lack of explicit theorising over the concept of "gender" even when gender biases are the primary concern of a paper @xcite ; and pointing to lack of considerations about the ethical implications of biases in nlp frameworks @xcite . in this paper, we build on these findings and contribute to ongoing work measuring and quantifying the effects of biases in nlp. we focus on one specific downstream task, namely named entity recognition (ner), and we focus only on the danish language. we examine error disparities as a function of sensitive features @xcite , where earlier work has shown differences across different demographic groups, namely gender and ethnicity @xcite . existing work has highlighted how unintended bias in nlp systems leads to systematic differences in performance for different demographic groups @xcite @xcite . in response to these results, various frameworks, fairness metrics, and recommendations for the field have been developed to quantify and mitigate bias @xcite @xcite @xcite . additionally, a growing body of work has demonstrated how counterfactual data augmentation (cda) of training data can be used to mitigate biases in nlp frameworks. this approach has been used for coreference resolution @xcite , and its applicability has been shown for a broader set of nlp tasks @xcite . we propose another use of data augmentation, namely as a method to test the robustness of nlp models and uncover potential social biases in the models. informed by intersectional feminism (crenshaw, 2013), we expand on earlier analysis to investigate the effect of different dimensions of bias and prejudice. the fundamental idea in intersectional feminism relates to how multiple dimensions of inequality result in complex, intersected inequality that cannot be accounted for through an isolated analysis of the single inequalities. for example, minority women might experience other types of discrimination than majority women and still others than those experienced by minority men. as the discussion and investigation of bias require more than a narrow focus on the overall performance score, adding nuances to bias tests opens up new findings and further reflections. in this paper, we examine how names mainly used by minority communities and names used by different genders affect the performance of ner models together. including non-gendered names in our experiment, we furthermore look to challenge the binary understanding of gender dominating the field of bias research. our experiments are limited to danish, a relatively high-resource language from a fairly homogeneous society with a restrictive gendered name law. our results demonstrate that for contemporary danish ner, error disparity is not evenly distributed across social groups and genders. this result adds significant nuances to the discussion of bias outlined in earlier iterations of this study @xcite by highlighting the importance of nuanced perspectives on performance scores @xcite and encouraging awareness of who is affected by nlp pipelines. in drawing attention to differences in performance across sensitive attributes, our focus is on biases as representational harms @xcite . the harmful aspects derive from the consequences of being excluded from the functionalities of automated systems employed in specific contexts. communities and individuals who are unrecognised risk falling into the residual space of being unseen and treated as irrelevant @xcite . in the case of textual cultural heritage, this manifests itself as archival silence, the absence of certain voices, stories, and histories @xcite . we argue that it is vital for those studying textual cultural heritage data with language technology to be able to measure the kinds of bias we outline, in order to avoid reproducing this silence. ## bias in nlp according to one influential definition, bias in computer systems can be defined as systems which 'systematically and unfairly discriminate against certain individuals or groups of individuals in favour of others' @xcite . this can be further broken down to distinguish between preexisting biases with roots in institutions, practices, and attitudes; technical biases arising from the resolution of issues in the technical specifications; and emergent bias which occurs in a usecontext after the implementation of a given system. it has furthermore been suggested to include 'freedom of bias' in the criteria for good computer systems. blodgett et al. (2020) provide a survey of bias research in nlp specifically and present a conceptual framework to characterize and compare biases. drawing on earlier work @xcite , they distinguish between allocation bias and representational bias. the former is a difference in the allocation of resources and opportunities; while the latter is differences in representations, such as stereotyping and negative generalisation of social groups. representational bias furthermore includes differences in system performance, such as how well an automated system performs for different demographic groups. we focus on representational bias as differences in system performance, measured as differences in error on a particular task. crawford (2017) emphasise representational bias as harmful in itself, mirroring the ideas of an emergent bias in @xcite . further bias emerges when systems whose performance differs systematically across different demographic groups are implemented. in social science and humanities, researchers who apply nlp tools in their work need to consider such performance differences when deciding which framework to use. for example, a researcher working in the field of gender history might need their models to be particularly robust with respect to gender; a scholar of social media might have a specific reason to require that their model is particularly robust to different ethnicities represented in their data. for those who work with cultural heritage data, there may therefore need to be a necessary trade-off between the overall accuracy of a particular framework and the bias that it exhibits relative to different groups. in the following section, we outline how existing societal biases in denmark make it crucial to test nlp frameworks for technical biases. ## results in more pertinently, our results show that the ner performance of every model is affected by the data augmentations. it is immediately apparent, though, that not all models are affected equally, and not all augmentations cause pronounced effects. our results seem to demonstrate that danish language models are relatively more robust to the impact of randomly changing women and men's names at the majority level. however, this is not the case for unisex names, where our results show that all danish nlp models are significantly worse at recognising these compared to gender-conforming names. similarly, randomly replacing names with minority names results in significantly worse performance for all models. this suggests that danish nlp models contain a greater relative bias regarding ethnicity than the binary gender division, emphasised by the results showing that all models performed consistently better for majority women's names than for minority men's names. for scandiner, all dacy models, danlp bert, flair, and nerda, the performance for minority women's names and minority men's names are similar -but still significantly lower than names from 'majority all'. for polyglot and the spacy models, the performance for minority women is worse than those for minority men. especially interesting are the results from dacy large, where there is no apparent bias for minority names if intersectionality is left out of the picture. however, a bias towards minority women is shown when minority names are divided into men's and women's names. scandiner performs overall best of all models, and even though it shows bias towards 'minority all' and 'minority men', it still outperforms dacy 11 see https://github.com/centre-for-humanitie s-computing/danish-ner-bias for code for the experimental pipeline large, which do not show the same bias in error rate for these groups. one could argue that for the best performing models, scandiner, and the dacy models, the differences in f1 scores are overall negligible. however, as small differences accumulate when used on large corpora we argue that even seemingly small differences (which are statistically significant) should be taken into consideration in an nlp pipeline. this becomes more pronounced when one considers the increase in error rate. the best performing model is scandiner shows a 7% increase in error from 'majority all' to 'minority women'. similarly, for dacy medium this amounts to an increase in error rate of 17%. for the poorest performing polyglot model, we calculate a 72% increase in error rate. hence, according to figure 1 , we conclude that danish nlp frameworks perform best for subgroups a and c (majority people). on the other hand, the models perform significantly worse for subgroups b and d (minority people), and some models are worst for subgroup b (minority women) specifically. adding unisex names and the gendered demarcation of the minority lists (see 5-7 in the list in section 3) to our tests shows that the error disparity is not evenly distributed across the social groups in figure 1 . these results open up the narrow focus on the overall performance scores and are significant contributions to the examination of bias started in earlier iterations of this study @xcite . ## discussion much of the work on representational bias focus on system performance and the concrete impact on individuals and groups as a result of biased models. however, we argue that similar considerations should underlie research applications of nlp, such as the use of language technology to study cultural heritage data. by ignoring the disparate performance of nlp frameworks on downstream tasks, we risk overlooking the testimony of marginalised voices in our corpora and archives. previous work has outlined how, in classification systems, residual categories are those that are left out when categories are established @xcite . by not complying with the agreed-upon categories, the 'other' fall between the cracks of the categorisation schema. this can happen if the object is too complicated to classify in the often taken-for-granted categories or if the residual is unknown to the system. falling into a residual space can result in people's experience being disregarded or overlooked, consciously or otherwise. in the context of named entity recognition, the classification performed is either recognised or unrecognised, and we argue that people whose names are unrecognised by automated systems reside in the residual spaces. our results show that, for contemporary danish ner, there are differences in performance along different demographic lines -differences that may not have been obvious without testing performance for the different subgroups. this, first and foremost, highlights the importance of challenging the narrow focus on overall performance score @xcite and sheds light upon the existence of diversity in who is affected. furthermore, these results also show a difference in the risk of residing into the residual space and potentially being disregarded and mistreated. the technical biases in these nlp frameworks risk reinforcing the existing structural biases if put into use. therefore, we recommend nlp practitioners to take accountability @xcite and consider these subgroup-specific performance results. the responsibility of measuring and mitigating such biases should be placed on those developing and implementing the tools -not on the marginalised group who are unfairly treated by the systems @xcite . in this work, we defined bias as the difference in error rate across different demographic subgroups. this bias is only tested for one specific task. for our data augmentation, we used the dane corpus, which consists of a diverse set of written and spoken danish from 1983-1992. however, minority names might occur more frequently in contexts which differ substantially from this corpus. if this is the case, our reported performances might vary according to how well danish nlp frameworks perform on ner for minority names 'in the wild'. hence, assessing the potential bias towards minority people might be even more complex. a similar issue arises when approximating ethnicity for social groups through the use of name lists. this approach leaves out minority people who take names typical for the majority group. however, when it comes to the performance of ner tools, minority people with majority names are not at the same risk of being unfairly treated by ner tools as people with minority names. the reverse is also the case: a person from the ethnic majority with a name typical for the minority is at greater risk of not being recognised by ner tools than people with majority names. nevertheless, this is not central to our analysis, insofar as we are only inferring at group level when examining the distribution of error rates across different social groups. further complexities in the use of names are the effect of rare names. for unisex names, we included the 500 most used names, which are approved unisex names in denmark. in this list, there are names that are common gendered names, such as 'anne', which in denmark is primarily used by women. if we filtered out common and primarily gendered names, the performance might be even poorer, but then it might be an effect of rare names rather than unisex names. nevertheless, this paper presents an innovative experimental method which adds nuanced perspectives to the overall performance evaluation for these models. based on data augmentation and the use of name lists as proxies for multiple dimensions of inequality, the method allows for an intersectional analysis of biases in danish nlp models used for named entity recognition. such findings are important to incorporate into scholarly pipelines in order to avoid enforcing archival silence. ## conclusions in this paper, we have shown the importance of intersectional analysis of biases in natural language processing (nlp) frameworks by testing danish nlp frameworks' robustness to data augmentation in named entity recognition (ner). by augmenting test data on gender-divided name lists for both majority and minority names, we have shown that danish nlp frameworks are relatively robust to the impact of women's and men's names at the majority level. however, all danish nlp models are significantly worse at recognising unisex names compared to gender-conforming names. furthermore, minority names cause significantly worse performance for all models. this suggests that danish nlp models contain a greater relative bias regarding ethnicity than the binary gender division. in the context of textual cultural heritage data, researchers regularly and increasingly incorporate language technology into their scholarly workflow. the most appropriate tool for a given task such as ner is usually chosen based on some pre-calculated metric score for how the technology performs for that task. however, based on the results presented here, we argue that a raw performance measure should not be the only criterion for deciding which nlp model to use. instead, we emphasise that, in the case of textual cultural heritage data, accuracy is not all you need. we encourage researchers to take these sub-group-specific performance measures into account when setting up their research pipeline.