Addressing Healthcare-related Racial and LGBTQ + Biases in Pretrained Language Models (2024)

Abstract: Recent studies have highlighted the issue of Pretrained Language Models (PLMs) inadvertently propagating social stigmas and stereotypes, a critical concern given their widespread use. This is particularly problematic in sensitive areas like healthcare, where such biases could lead to detrimental outcomes. Our research addresses this by adapting two intrinsic bias benchmarks to quantify racial and LGBTQ+ biases in prevalent PLMs. We also empirically evaluate the effectiveness of various debiasing methods in mitigating these biases. Furthermore, we assess the impact of debiasing on both Natural Language Understanding and specific biomedical applications. Our findings reveal that while PLMs commonly exhibit healthcare-related racial and LGBTQ+ biases, the applied debiasing techniques successfully reduce these biases without compromising the models’ performance in downstream tasks.

## introduction pretrained language models (plms) have significantly advanced the field of natural language processing (nlp), achieving state-of-the-art results across diverse applications. their integration into healthcare contexts, ranging from clinical note interpretation @xcite to medical dialogue summarization @xcite and radiology report analysis @xcite , has been particularly noteworthy. however, the impressive performance of plms is marred by inherent social biases due to their training on extensive and varied datasets. these biases, encompassing racial, gender, and religious prejudices @xcite @xcite , become * co-corresponding authors. especially concerning in high-stakes domains like healthcare. in such settings, biased plms can lead to unfair and potentially harmful outcomes @xcite . studies like @xcite and @xcite highlight the detrimental effects of these biases, such as biased clinical decisions and the perpetuation of harmful stereotypes. to effectively tackle the challenge of ingrained biases in plms, researchers have introduced various bias-measuring techniques and datasets aiming to quantify and benchmark these biases @xcite @xcite . concurrently, several debiasing methods have been developed, focusing on either mitigating biases in model outputs or eradicating latent biases within the models themselves @xcite @xcite @xcite . despite these advancements, current bias benchmarks fall short in specifically measuring harmful stereotypes in healthcare, as exemplified in figure 1 . furthermore, the efficacy of existing debiasing methods in addressing healthcare-related biases in plms remains unexplored. this paper aims to fill this gap by examining latent racial and lgbtq+ biases in plms, particularly those manifesting as stereotypical associations with diseases, conditions, and assumptions based on race and sexual orientation. drawing from medical literature, we identify prevalent stereotypes among both the general public and medical professionals, adapt existing bias benchmarks for this context, and apply debiasing techniques to assess their effectiveness in eliminating these specific biases. finally, we conduct comprehensive experiments to assess the impact of these debiasing interventions on the language modeling capabilities of plms. in this work, we present three key contributions: lgbtq+ biases in plms. our experiments encompass 15 popular plms, and we detail the extent of bias identified in each. lgbtq+ biases in plms. the effectiveness of these methods is thoroughly evaluated and reported. implicit racial and lgbtq+ biases in healthacare. extensive research has demonstrated that implicit biases among healthcare professionals significantly influence their treatment decisions, leading to disparities across different patient demographics (hall et al., 2015; maina et al., 2018). for instance, moskowitz et al. (2012) revealed a prevalent implicit association of african americans with conditions like obesity and hypertension among physicians, adversely impacting patient care. similarly, a tendency among physicians to underrate the competence of black patients, influencing prescription practices, was noted (fitzgerald and hurst, 2017). the lgbtq+ community faces notable healthcare disparities rooted in societal biases (fingerhut and abdou @xcite , such as the persistent prejudice among healthcare providers that transgender people are mentally ill @xcite . given recent findings @xcite @xcite that plms can inherit human-like biases, this work aims to quantify healthcare-related biases in plms, focusing on harmful stereotypes and stigmas affecting marginalized groups. we adapt existing bias benchmarks to measure implicit associations in plms between certain demographics (e.g., white/black or cis/lgbtq) and stereotypical diseases, along with healthcare-related stigmas and assumptions linked to these groups. quantifying bias in plms. the exploration of bias and stereotypes in plms, particularly within the healthcare domain, remains underdeveloped. this gap is partly due to the current reliance on benchmarks composed of specialized datasets and specific metrics tailored for those datasets. for instance, @xcite introduced the stere-oset dataset and a corresponding method to evaluate plm biases through the preference for stereotypical sentences. while this sentence preference approach is adaptable to different contexts, the fixed dataset limits the scope of bias analysis to predefined instances. similarly, traditional weat tests, as proposed by @xcite , face challenges in generalizing to diverse bias forms due to the vocabulary limitations of the original dataset. recent efforts @xcite @xcite have expanded weat by incorporating a wider range of biases and contextualizing sentences, thus broadening the scope of analysis. existing bias benchmarks @xcite @xcite predominantly focus on gender and racial biases in social settings @xcite or occupational biases @xcite . while the dataset provided in nadeem et al. (2020) covers a broad spectrum of stereotypical and anti-stereoypical examples, the majority of these instances are situated outside of the healthcare context. similarly, although the crowd-sourced dataset in nangia et al. (2020) presents a robust methodology for bias measurement, this dataset does not focus on stereotypes and stigmas prevalent in the healthcare domain. in §3.1, we describe how we tailor our approach to generate examples within the healthcare domain and quantify biases by adapting the benchmarks and strategies of nadeem et al. (2020); nangia et al. (2020). recent research on debiasing plms such as meade et al. (2022) assesses the effectiveness of debiasing methods by utilizing datasets in nadeem et al. (2020); nangia et al. (2020). therefore, their results do not demonstrate the effect of removing healthcare-related biases from plms. zhang et al. (2020) examines the impact of bias in healthcarerelated tasks but only for a single bert model, without considering debiasing effects. felkner et al. (2022, 2023) introduced the winoqueer dataset that addresses social stereotypes regarding the lgbtq+ community and evaluated the effectiveness of removing such lgbtq+ related biases in plms. however, similar to meade et al. (2022), felkner et al. (2022, 2023) 's debiasing results using the winoqueer dataset do not reflect biases in plms regarding healthcare-specific lgtbtq+ stigmas and biases, particularly those involving disease assumptions. while these prior works contribute significantly in their findings and methodologies, they address only a fraction of the broader issue our work aims to tackle. to thoroughly investigate our research questions, we synthesize methods and approaches from these studies. we first adapt existing benchmark datasets and their metrics to measure the specific biases we focus on. we then conduct extensive experiments with a diverse set of popular plms to assess the extent of bias and the efficacy of debiasing techniques. finally, we analyze the impact of these debiasing efforts on downstream biomedical tasks, providing a comprehensive evaluation of bias mitigation in plms within the healthcare domain. ## debiasing techniques our study critically evaluates the efficacy of three prominent debiasing techniques: dropout (srivastava et al., 2014), sentencedebias @xcite , and iterative nullspace projection (inlp) @xcite , as applied to pretrained language models (plms). the outcomes of these evaluations are detailed in ## conclusion in this study, we have developed benchmarks to effectively quantify healthcare-related racial and lgbtq+ biases present in widely utilized pretrained language models. our findings reveal a consistent presence of biases in these plms, manifested through implicit associations between marginalized demographics and stereotypical diseases or harmful stigmas within healthcare contexts. additionally, we have conducted an empirical analysis of various debiasing techniques applied to plms, including dropout, sentencedebias, and iterative nullspace projection. our results indicate that sentencedebias generally emerges as the most effective method for reducing biases. crucially, when applying these debiased models to several downstream tasks, we observe that popular debiasing techniques do not significantly compromise the performance of the models. this outcome underscores the feasibility of implementing debiasing measures in plms without sacrificing their functional efficacy. we fully recognize that our definitions and methods may be considered narrow by some. we do not intend to speak for any community or demography that has suffered from disparate treatments in healthcare-related settings. specifically, in addition, we acknowledge that the presence of certain biases within plms are medically necessary, aiding both models and physicians in making accurate decisions. on the other hand, there also exist, in plms, biases stemming from stigmas and ill-informed stereotypes that pose undue influence on model decisions and therefore require mitigation. although our research aims to comprehensively identify biases in llms, determining whether certain biases are medically necessary or unnecessary is beyond our study's scope and should be left to medical professionals. the objective of this research is not to make those particular differentiations; rather, it focuses on identifying biases and stereotypes and exploring the ramifications of removing this information on plms. we do not endorse any of the offensive stereotypes used as examples to demonstrate methodology in the paper. it is the sincere hope of the authors of this paper that our work will not only serve to identify stereotypical biases in plms but also offer insight into reducing them for plms' safe and ethical usage.