Bias assessment for experts in discrimination, not in computer science (2023)

Abstract: Approaches to bias assessment usually require such technical skills that, by design, they leave discrimination experts out. In this paper we present EDIA, a tool that facilitates that experts in discrimination explore social biases in word embeddings and masked language models. Experts can then characterize those biases so that their presence can be assessed more systematically, and actions can be planned to address them. They can work interactively to assess the effects of different characterizations of bias in a given word embedding or language model, which helps to specify informal intuitions in concrete resources for systematic testing.

## introduction machine learning models and data-driven systems are increasingly being used to support decisionmaking processes. such processes may affect fundamental rights, like the right to receive an education or the right to non-discrimination. it is important that models can be assessed and audited to guarantee that such rights are not compromised. ideally, a wider range of actors should be able to carry out those audits, especially those that are knowledgeable of the context where systems are deployed or those that would be affected. several studies found that linguistic representations learned from corpora contain associations that produce harmful effects when brought into practice, like invisibilization, self-censorship or simply as deterrents @xcite . the effects of these associations on downstream applications have been treated as bias, that is, as systematic errors that affect some populations more than others, more than could be attributed to a random distribution of errors. this biased distribution of errors results in discrimination of those populations. unsurprisingly, such discrimination often affects negatively populations that have been historically marginalized. to detect and possibly reduce such harmful behaviour, many techniques for measuring and mitigating the bias encoded in word embeddings and large language models (llms) have been proposed by nlp researchers and machine learning practitioners @xcite . in such works social scientists have been mainly reduced to the ancillary role of providing data for labeling, rather than being considered as core team @xcite . current audits of data-driven systems often require technical skills that are beyond the capabilities of most of the people with knowledge on discrimination. the technical barrier has become a major hindrance to engaging experts and communities in the assessment of automated systems. moreover, we think approaching social risk mitigation through algorithmic calculations or adjustments is reductionist. we believe the part of the process that can most contribute to bias assessment are not subtle differences in metrics or technical complexities incrementally added to existing approaches, as is the case of a good portion of academic work in the area. instead, we believe what can most contribute to an effective assessment of bias in nlp is precisely the linguistic characterization of the discrimination phenomena @xcite . that is why our aim with this work is to open up the participation of experts both on the complexities of the social world and on communities that are being directly affected by ai systems. participation would allow processes to become transparent, accountable, and responsive to the needs of those directly affected by them. the rest of this paper is organized as follows. in the next section we state the principles for integrating discrimination experts in the bias assessment process. we then review the shortcomings of some approaches to bias assessment, and argue for the need for a tool specifically targeted to facilitate the integration of non-technical persons in the process of bias assessment. then, we describe edia, the tool we developed to address this need. we finish with a discussion of our experiences in handson sessions with discrimination experts using the tool. appendices with more extensive descriptions of the tool and a user story are also provided. a demo of edia can be used at https://huggingface.co/spaces/ vialibre/edia , allowing to explore the word2vec from spanish billion word corpus embedding @xcite and beto @xcite as the default language model. the tool is available at https: //github.com/fvialibre/edia , and can be instantiated to explore different word embeddings and language models, independently of language, as is showcased in the colab jupyter notebook illustrating the functionalities of edia 1 . ## a critical perspective on methods for bias assessment in the last years the academic study of biases in language technology has been gaining growing relevance, with a variety of approaches accompanied by insightful critiques. early work on bias focused on finding metrics that allowed to adequately assess bias in word embeddings (i.e. @xcite ; @xcite ). most of the following work focused on technical subtleties about metrics, extensions to other languages or contexts, application to language models, evaluation on downstream tasks or automating the whole process, from assessment to mitigation @xcite @xcite . ## an intuitive tool to explore bias this section provides a description of edia (acronym for the spanish of stereotypes and discrimination in artificial intelligence), a visual interface framework for the analysis of bias in word embeddings and in llms foot_0 . a more detailed description of the tool can be seen in @xcite . edia follows the design principles stated in 2, trying to fill a gap in the landscape of existing frameworks for bias assessment. it provides four main functionalities: exploring the learning data, exploring the distribution of words in an embedding space, systematizing biases in words and exploring biases in sentences. in what follows we describe these functionalities. in appendix a we describe a user story showcasing how this tool may be used. framework reference word language requieres mitigation counterfactuals embeddings models nlp techniques analysis analysis analysis knowledge implemented wordbias ghai et al. (2021) ✓ ✗ ✗ ✗ ✗ verb rathore et al. (2021) ✓ ✗ ✓ ✓ ✗ lit tenney et al. (2020) ✓ ✓ ✓ ✗ ✓ edia benotti et al. (2023) ✓ ✓ ✗ ✗ ✗ ## discussion we have argued that bias is a complex phenomenon that needs to be addressed with specific expertise, or else risk a reductionist approach. such approaches have been shown to produce inadequate results. to our knowledge, existing tools to address bias require technical expertise of different kinds. such requirement will probably hinder the involvement of discrimination experts in the bias assessment problem, specially those experts belonging to minorized communities or in the global south. we have developed a tool, edia, that eliminates unnecessary technicalities. the main aim of edia is to facilitate that discrimination experts can build the linguistic resources (word lists and word sentences) that are the keystone of bias assessment by interacting with the relevant word embeddings and language models. we have worked with a variety of discrimination experts in four hands-on workshops, two before the development of the graphical interface of the tool, and two after an initial prototype, involving 70 and 30 experts on diverse fields and aspects of discrimination. experts worked in their area of expertise, and successfully modeled different biases, including ageism, fatphobia, ableism or aporafobia. they also explored stereotypes associated to the province of origin within argentina, gender violence, the young or different psychological features. participants were satisfied and we are planning to carry out a second phase of the project where we expect them to produce linguistic resources resulting from their systematic exploration of those biases. we have also carried out hands-on sessions with general public, not discrimination experts, and they have been able to use edia to intuitively explore biases in language models and consolidate a critical perspective on those technologies. the work presented here is just the starting point of a much longer endeavor. our vision is that firms and institutions integrate this kind of exploration within the development of language technologies, engaging discrimination experts as a permanent asset in their teams, well before deploying any product. we would also like the general population to carry out this kind of audits, and that this is part of a more aware, empowering technology education for all. we are also working toward building a repository of linguistic resources that represent different biases, as characterized by different communities and in different contexts. ## limitations for the development of our tool edia we designed three workshops with 50 people each in which we received feedback about its usability. we based our decisions on the feedback we received from different experts in discrimination in hands-on experiences using early prototypes of the tool. most of the experts worked on gender discrimination and other kinds of discrimination are less represented in our workshops. for more detail on the workshops we conducted with users to assess design decisions and the overall accessibility of the methodology, see @xcite . we did not ensure that the participants in our workshops were representative of the intended population, although we did our best efforts to have people with diverse backgrounds in social and objectives. although we did our best to have a diverse team, including social scientists, communicators, linguists and computer scientists, of diverse backgrounds, ages and geographical origins, we could not manage to integrate people with disabilities, or without university education. our workshops were conducted in spanish. our tool works for english too but the evaluation and the design was only evaluated for spanish. we do not provide mitigation strategies in our tool, we only make bias assessment available for not experts. our assessment of bias in word embeddings is limited to a binary representations of bias. we allow for a more nuanced analysis of biases by combining two binary biases, characterized by four extremes (feminine vs masculine, old vs young, etc), as displayed in figure 4 . the assessment of bias in large language models, through sentences, overcomes this limitation. ## ethical considerations our tool can benefit researchers from social sciences that want to study biases in word embeddings or language models. it can also be used by small companies that cannot train their own language models and that want to study the biases present in different pre trained language models when deciding which to use in their products. the metrics we use to measure bias are known to have limitations @xcite and the benchmarks existing in the area @xcite . a potential risk of our tool is that users assume that our tool can be used to show that a model is not biased in a particular dimension without considering the limitations of the metrics and the benchmarks. finally, this work discusses how to involve discrimination experts in the exploration of biases in nlp and argues that this is important. this might discourage researchers in nlp working on bias analysis and mitigation to keep working in this area because they do not have access to interdisciplinary experts. in this way, we could discourage work in an area we believe is important. we think different approaches are valuable in this area and studying in more detail the metrics of the area is very important and needs deeper technical expertise. this might not require discrimination experts if reliable benchmarks are available in the area. participation in our workshops involved answering a pre-survey, a post-survey, and a 3-hour hands-on in-person workshop. participants were volunteers and did not receive compensation. edia does not censor the models, so words that might be censored by other tools can be explored. in one of our workshops the participants explored words associated to feminine sexuality vs words associated with masculine sexuality and found that feminine words were associated with disease while sexual masculine words were associated with health in the language model @xcite . in natural language processing, pages 69-75, florence, italy. association for computational linguistics. archit rathore, sunipa dev, jeff m. phillips, vivek srikumar, yan zheng, chin-chia michael yeh, junpeng wang, wei zhang, and bei wang. 2021. verb: visualizing and interpreting bias mitigation techniques for word representations. joão sedoc and lyle ungar. 2019. the role of protected class word lists in bias identification of contextualized word representations. in proceedings of the first workshop on gender bias in natural language processing, pages 55-61, florence, italy. association for computational linguistics. ian tenney, james wexler, jasmijn bastings, tolga bolukbasi, andy coenen, sebastian gehrmann, ellen jiang, mahima pushkarna, carey radebaugh, emily reif, and ann yuan. 2020. the language interpretability tool: extensible, interactive visualizations and analysis for nlp models. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 107-118, online. association for computational linguistics. james wexler, mahima pushkarna, tolga bolukbasi, martin wattenberg, fernanda viégas, and jimbo wilson, editors. 2019. the what-if tool: interactive probing of machine learning models. jieyu zhao, tianlu wang, mark yatskar, ryan cotterell, vicente ordonez, and kai-wei chang. 2019. gender bias in contextualized word embeddings. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 629-634, minneapolis, minnesota. association for computational linguistics. pei zhou, weijia shi, jieyu zhao, kuan-hao huang, muhao chen, ryan cotterell, and kai-wei chang. 2019. examining gender bias in languages with grammatical gender. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5276-5284, hong kong, china. association for computational linguistics. in this section we describe a user story that presents a paradigmatic process of bias exploration and assessment. we would like to note that this user story was originally developed to be situated in argentina, the local context of this project. it was distilled from experiences with data scientists and experts in discrimination that are described in @xcite . however, in order to make understanding easier for non-spanish speaking readers, we adapted the case to work with english, and consequently localized the use case as if it had happened in the united states. the users. marilina is a data scientist working on a project to develop an application that helps the public administration to classify citizens' requests and route them to the most adequate department in the public administration office she works for. tomás is a social worker within the non-discrimination office, and wants to assess the possible discriminatory behaviours of such software. the context. marilina addresses the project as a supervised text classification problem. to classify new texts from citizens, they are compared to documents that were manually classified in the past. new texts are assigned the same label as the document that is most similar. calculating similarity is a key point in this process, and can be done in many ways: programming rules explicitly, via machine learning with manual feature engineering or by deep learning, where a key component is word embeddings. marilina observes that the latter approach has the least classification errors on the past data she separated for evaluation (the so called test set). moreover, deep learning seems to be the preferred solution these days, it is often presented as a breakthrough for many natural language processing tasks. so marilina decides to pursue that option. an important component of the deep learning approach she uses are word embeddings. marilina decides to try a well-known word embedding, pretrained on wikipedia content. when she integrates it in the pipeline, there is a boost in the performance of the system: more texts are classified correctly in her test set. looking for bias. marilina decides to look at the classification results beyond the figures of classification precision. being a descendant of latin american immigrants, she looks at documents related to this societal group. she finds that applications for small business grants presented by latin american immigrants or citizens of latin american descent are sometimes erroneously classified as immigration issues and routed to the wrong department. these errors imply a longer process to address these requests in average, and sometimes misclassified requests get lost. in some cases, this mishap makes the applicant drop the process. finding systematic errors. intrigued by this behaviour of the automatic pipeline, she makes a more thorough research into how requests by immigrants are classified, in comparison with requests by non-immigrants. as she did for latin american requests, she finds that documents presented by other immigrants have a higher error rate than the non immigrants requests. she suspects that other societal groups may suffer from higher error rates, but she focuses on latin american immigrants because she has a better understanding of the idiosyncrasy of that group, and it can help her establish a basis for further inquiry. she finds some patterns in the misclassifications. in particular, she finds that some particular business, like hairdressers or bakeries, accumulate more errors than others. finding the component responsible for bias. she traces the detail of how such documents are processed by the pipeline and finds that they are considered most similar to other documents that are not related to professional activities, but to immigration. the word embedding is the pipeline component that determines similarities, so she looks into the embedding with the edia toolkit foot_1 . she defines a bias space with "latin american" in one extreme and "north american" in the other, and checks the relative position of some professions with respect to those two extremes, as can be seen in figure 6 , on the left. this graph is generated using the button called "find 2 stereotypes" in the tab. she finds that, as she suspected, some of the words related to the professional field are more strongly related to words related to latin american than to words related to north american, that is, words like "hairdresser" and "bakery" are closer to latin american. however, the words more strongly associated to north american do not correspond to her intuitions. she is at a loss as to how to proceed with this inspection beyond the anecdotal findings, and how to take action with respect to the findings. that is when she calls for help to the non-discrimination office. assessing harm. the non-discrimination office appoints tomás for the task of assessing the discriminatory behavior of the software. briefed by marilina about her findings, he finds that misclassifications do involve some harm to the affected people that is typified among the discriminatory practices that the office tries to prevent. misclassification implies that the process takes longer than for other people, because they need to be reclassified manually before they can actually be taken care of. sometimes, they are simply dismissed by the wrong civil servant, resulting in unequal denial of benefits. in many cases, the mistake itself has a negative effect on the self-perception of the issuer, making them feel less deserving and discouraging the pursuit of the grant or even the business initiative. tomás can look at the output of the system, but he cannot see a rationale for the system's misclassifications, he doesn't know how the automatic classification works. detecting the technical barrier. tomás understands that there is an underlying component of the software that is impacting in the behaviour of classification. marilina explains to him that it is a pretrained word embedding, and that a word embedding is a projection of words from a sparse space where each context of co-occurrence is one of thousands of dimensions into a dense space where there are less dimensions, obtained with a neural network. she says that each word is a vector with numbers in each of those dimensions. tomás feels that understanding the embedding is beyond his capabilities. then marilina explains to him that words are represented as a summary of their contexts of occurrence in a corpus of texts, but this cannot be directly seen, but explored using similarity between words, so that more similar words are closer. finding an intuitive tool for bias exploration. she shows him some of the tools available to assess bias in the edia demo foot_2 , which do not require tomás to handle any programming or seeing any code. marilina resorts to the available introductary materials for our tool to explain bias definition and exploration easily to tomás using the "biases in words" tab. he quickly grasps the concepts of bias space, definition of the space by lists of words, assessment by observing how words are positioned within that space, and exploration by modifying lists of words, both defining the space and positioned in the space using the "explore words" tab with words that he know are representative for their domain. he gets more insights on the possibilities of the techniques and on possible misunderstandings by reading examples and watching the short tutorials that can be found with the tool. he then understands that word ambiguity may obscure the phenomena that one wants to study if exploring single words, that word frequency has a big impact, and that language-specific phenomena, like grammatical gender or levels of formality, need to be carefully taken into account. he uses the tab "biases in sentences" when words are highly ambiguous or when he needs to express a concept using multiword expressions such as in "latin america". after some toying with the demo, tomás believes this tool allows him to adequately explore biases, so marilina deploys a local instance of the tool, which will allow tomás to as-sess the embedding that she is actually using in her development, and the corpus it has been trained on. explore the corpus behind the embeddings. to begin with, tomás wants to explore the words that are deemed similar to "latin american", because he wants to see which words may be strongly associated to the concept, besides what marilina already observed. he uses the "data tab" of edia, described in section 4 to explore the data over which the embedding used by marilina has been inferred. he finds that the embedding has been trained with texts from newspapers. most of the news containing the word latin american deal with catastrophes, troubles and other negative news from latin american countries, or else portray stereotyped latin americans, referring to the typical customs of their countries of origin rather than to their facets as citizens in the united states. with respect to business and professions, latin americans tend to be depicted in accordance with the prevailing stereotypes and historic occupations of that societal group in the states, like construction workers, waiters, farm hands, etc. he concludes that this corpus, and, as a consequence, the word embedding obtained from it, contains many stereotypes about latin americans which are then related to the behaviour of the classification software, associating certain professional activities and demographic groups more strongly with immigration than with business. marilina says that possibly they will have to find another word embedding, but he wants to characterize the biases first so that he can compare to other word embeddings. formalize a starting point for bias exploration. tomás builds lists of relevant words, with the final objective to make a report and take informed action to prevent discriminatory behavior. first, he builds the sets of words that will be representing each of the relevant extremes of the bias space. he realizes that marilina's approach with only one word in each extreme is not quite robust, because it may be heavily influenced by properties of that single word. that is why he defines each of the extremes of the bias space with longer word lists, and experiments with different lists and how they determine the relative position of his words of interest. words of interest are the words being positioned in the bias space, words that tomás wants to characterize with respect to this bias because he suspects that their characterization is one of the causes for the discriminatory behavior of the classification software. to find words to include in the word lists for the extremes, tomás resorts to the functionality of finding the closest words in the embedding. using "latin american" as a starting point, he finds other similar words like "latino", and also nationalities of latin america using the "explore words" tab. he also explores the contexts of his words of interest. doing this, he finds that "shop" occurs in many more contexts than he had originally imagined, many with different meanings, for example, short for photoshop. this makes him think that this word is probably not a very good indicator of the kind of behavior in words that he is trying to characterize. he also finds that some professions that were initially interesting for him, like "capoeira trainer" are very infrequent and their characterization does not have a correspondence with his intuition about the meaning and use of the word, so he discards them. finally, he is satisfied with the definition provided by the word lists that can be seen in figure 6 , right. with that list of words, the characterization of the words of interest shows tendencies that have a correspondence with the misclassifications of the final system: applications from hairdressers, bakers, dressmakers of latino origin or descent are misclassified more often than applications for other kinds of businesses. even though they are assessing biases in a word embedding, that represents words in isolation, collapsing all senses of a word, tomás believes that once they are characterizing this bias, they may best take advantage of the effort and also build a list of sentences characterizing the same bias, to be used when assessing this same bias in a language model, for example, to assess the behavior of a chatbot. to provide him with inspiration, marilina offers tomás a benchmark for bias exploration developed for english and french @xcite and tomás uses that dataset partially to define his own list of sentences to explore relevant biases in this domain. report biases and propose a mitigation strategy . with this characterization of the bias, tomás can make a detailed report of the discriminatory behavior of the classification system. from the beginning, he suspected the cultural and social reasons behind the errors, which affect more often people of latin american descent applying for subsidies for a certain kind of business. however, his intuitive manipulation of the underlying word embedding allowed him to find words and phrases that give rise to the pattern of behavior he was observing, going beyond the cases that he has actually been able to see as misclassified by the system, and predicting other cases. moreover, understanding the pattern of behavior allowed him to describe properties of the underlying corpus that would be desirable in order to find another word embedding. he can propose strategies like editing the sentences containing hairdressers, designers and bakers to show a more balanced mix of nationalities and ethnicities in them. finally, he has a list of words and sentences that can give marilisa to measure and compare the biases with respect to these aspects in other word embeddings