Sociodemographic Bias in Language Models: A Survey and Forward Path (2024)

Abstract: Sociodemographic bias in language models (LMs) has the potential for harm when deployed in real-world settings. This paper presents a comprehensive survey of the past decade of research on sociodemographic bias in LMs, organized into a typology that facilitates examining the different aims: types of bias, quantifying bias, and debiasing techniques. We track the evolution of the latter two questions, then identify current trends and their limitations, as well as emerging techniques. To guide future research towards more effective and reliable solutions, and to help authors situate their work within this broad landscape, we conclude with a checklist of open questions.

## introduction language models (lms) have demonstrated impressive performance across diverse tasks @xcite @xcite . however, much work reveals that lms adopt biases present in training data @xcite @xcite . sociodemographic bias has been defined to occur when a model performs differently across social groups @xcite . when lms are used in real-world applications, sociodemographic bias has the potential for negative societal impacts @xcite @xcite . with increasingly widespread usage, the urgency to understand and mitigate bias has grown. fig. 1 shows an increasing rate of publications on lm bias over the past decade, sourced from scopus. our survey synthesizes results from this rapidly growing area into a roadmap for future investigations. other surveys on bias in nlp have thoroughly examined a particular aspect of bias, such as methods for measuring bias @xcite , or identification of gender bias figure 1 : this graph shows number of papers/articles published each year (from 2013 to 2023) in scopus that contain the term 'bias' and ('nlp' or 'language models') in the title, abstract, or keywords. @xcite . unlike previous surveys, we provide a typology of works on bias over the past decade. further, we build upon foundational issues identified by @xcite by delving more deeply into methodological limitations, such as reliability issues. we also follow the recommendations of @xcite in consulting interdisciplinary approaches to improve the understanding of social bias. thus we begin the survey with a discussion of psychosocial perspectives on benefits versus harms of bias. our survey offers an up-todate understanding of a topic that has been garnering increasing interest. early in this literature, the development of bias mitigation or debiasing methods had questionable success; we argue that recent work using expert models during training shows particular promise. we conclude with a checklist of key questions that have continued to be challenging, to help steer future studies toward more effective and reliable methods, well-situated within the landscape of work on bias. we present a synthesis of works on bias through three perspectives: 1) a taxonomic categorization, 2) an evolutionary timeline, and 3) a roadmap for future work. we categorized the surveyed works into three major strands of investigation, as shown in fig. 2 : types of bias, quantifying bias, and debiasing techniques. we then organize the findings within each category and subcategory of our taxonomy. in addition, we examined the evolution over the past decade of techniques for bias measurement and bias mitigation, as shown in fig. 3 . this perspective separates trends that had a brief life from those that continue to have promise. while lm bias measurement and mitigation are critical for progress towards equitable lms, understanding the potential for harm is deeply intertwined with social factors outside the scope of nlp proper. thus we precede the presentation of the major types of bias research with a discussion of psychosocial perspectives (cf. @xcite . this is followed by a section describing our process for identifying candidate works, and our resulting typology where we place most of the surveyed works. sections 4-6 present limitations, the checklist and future directions. ## understanding bias interdisciplinary approaches to understanding bias as a psychosocial phenomenon have been argued to be important for clarifying how social harms might arise. research into human cognition and social behavior can provide valuable insights on sociodemographic bias in lms, as well as assessment of their potential for harm. for instance, research in psychology has long addressed the origins and expressions of social bias @xcite recent studies have begun to integrate ideas from psychology with nlp to better understand bias @xcite @xcite , showcasing the usefulness of interdisciplinary approaches. for example, research in psychology proposes that reduction of social bias can be achieved by engaging with individuals from diverse groups @xcite . a similar idea is reflected in @xcite , which advocates for lm engineers to reduce bias through engagement with people who might be affected by applications that use lms. one of the early works on quantifying bias -weat @xcite used the implicit association test from psychology @xcite to develop a foundation bias metric for lms. the stereotype content model (scm), a framework from social psychology, categorizes stereotypes into interpersonal and intergroup interactions, providing insights into bias dynamics @xcite . it proposes that human stereotypes are cap-tured by two social dimensions: warmth (e.g., trustworthiness, friendliness) and competence (e.g., capability, assertiveness). a recent work by @xcite used the scm framework to develop a bias mitigation method that generalizes across multiple social attributes, rather than one at a time. the nobel prize-winning psychologist and behavioral economist, daniel kahneman, argues that mental shortcuts (biases) are advantageous in situations requiring quick judgments @xcite . for example, due to bias based on strong knowledge priors, the sentence "a large mouse climbed over a small elephant" will immediately call to mind the appropriate relative sizes; to counter this assumption would require extra information @xcite . extrapolating kahneman's argument to nlp, bias based on common-sense knowledge could be advantageous in enhancing an lm's understanding of relations among real-world entities. this argues for the potential benefit of certain kinds of bias. @xcite defines disadvantageous bias as "the tendency to make systematic errors in judgment or decisions based on factors that are irrelevant or immaterial to the task at hand" and cautions that human judgment is susceptible to bias from irrelevant factors. turning to lm bias, we find previous nlp work aligned with kahneman's perspective in definitions of representational harm (crawford, 2017) and alloted harm @xcite . representational harm arises when an nlp system represents some social groups in a less favorable light than others. allotted harm arises when a system allocates resources or opportunities unfairly to a social group @xcite . in conclusion, ideas from psychology and behavioral economics provide a more informed understanding of bias. while some biases might contribute positively to model performance, others can have detrimental societal effects. an interdisciplinary approach would not only enrich our theoretical understanding of bias but could also guide the development of more effective methods to identify undesirable lm bias and lessen social harm. ## categories of work on bias in lms we used two strategies to identify candidate papers for our survey: 1) using the keywords "bias", "fair" and "fairness", we searched for papers in the acl anthology, neurips proceedings, facct, and aies conferences; 2) we included papers from citation graphs for retrieved papers. we examined papers released before january 1, 2024, and included them only if they addressed language modeling, thus omitting papers on speech, where different issues arise. these criteria narrowed down an initial large set of 308 papers to 273. we categorized the literature into three key areas. fig. 2 illustrates our taxonomy. we came up with this organization while iteratively reviewing all papers, and we believe it effectively captures the main trends in the field. due to the rapidly evolving field of lms, some upcoming studies may not fit neatly into these categories. to address this, we plan to release our literature repository publicly and update it regularly with the latest research. our work summarizes all 273 surveyed papers to provide a comprehensive understanding. due to space constraints, we couldn't cite all 273 works in the main body. to maximize coverage within the page limits, we selected at least two papers from each line of research depicted in figure 2 to be part of the main paper. in some cases, we wanted to cite more works but had to remove them due to space limitations. we apologize for any relevant works missed in the main body and have included a comprehensive list of all 273 papers in the appendix. ## limitations of current approaches the works surveyed here offer valuable insights towards understanding bias in lms, and demonstrating many innovative approaches and methodologies that have advanced the field. alongside the commendable progress, however, a thorough analysis of the body of work on bias reveals limitations. reliability issues with bias metrics. the robustness of existing bias metrics is questionable. metrics introduced in works within q1 and q3 change significantly, given minor changes in datasets or evaluation settings @xcite @xcite . similarly, template-based methods are highly sensitive to small modifications to words used in the templates @xcite @xcite . use of identical templates across bias categories. many of the works using template-based approaches @xcite use the same templates to assess diverse social biases, without considering whether certain template features should be specific to distinct types of bias. this approach risks conflating bias scores across categories, suggesting a need for more tailored templates to measure specific social biases accurately. alternatively, investigation of ways to generalize across templates to a more abstract approach, as in omrani sabbaghi et al. (2023), holds promise. limited scope of template-based bias measurement. template-based methods often use a restricted range of templates and target words, for example, focusing on us-based names for targets. additionally, these approaches suffer from author bias, as templates are manually designed by the authors @xcite . gap in translating bias metrics to real-world effects. there is a notable disconnect between bias metrics and their implications for real-world applications. bias metrics in q1 have been claimed to correlate poorly with real-world biases @xcite . @xcite argued that intrinsic and extrinsic metrics do not correlate with each other. such observations underscore the need for improvements in metric robustness and interpretability. weaknesses in finetuning approaches for debiasing. the majority of recent works on lm debiasing focus on finetuning, valued mainly for its simplicity and adaptability. however, its effectiveness is often questionable @xcite . the complexity and size of modern llms, which require extensive data, time, and resources to train, make it particularly challenging to eliminate bias through finetuning. further, these methods treat symptoms rather than root causes of bias, adjusting model outputs to appear less biased without actually removing bias from models @xcite . remarkably, some debiasing techniques can potentially increase bias @xcite . the absence of reliable bias metrics complicates the evaluation of the effectiveness of debiasing methods. we recommend that future works utilize a variety of metrics to thoroughly assess debiasing results. overemphasis on gender bias. lack of sociotechnical understanding of bias. the nlp literature exhibits little attention to the sociotechnical impacts of bias @xcite . similarly, there can be incomplete consideration of the complexity of sociodemographic bias @xcite . interdisciplinary collaborations could offer more nuanced insights and improved methodologies to measure, mitigate, prevent, and assess harms from bias. ## checklist a checklist can assist future work to avoid the pitfalls of previous work and build more effective and reliable measurement and debiasing methods across more types of sociodemographic bias. we present 13 questions divided into three categories. questions 1-6 focus on bias measurement (qb); questions 7-8 focus on bias mitigation (bm); questions 9-13 are applicable to all works on lm bias. we hope that future work guided by these questions can help authors situate their results within the broader literature on sociodemographic bias. despite their growing popularity, template-based methods for measuring bias face challenges @xcite . we believe that these challenges can be tackled with careful consideration of the limitations, such as lack of robustness, leading to more effective and reliable bias measurement. we anticipate that prompt-based methods will gain prominence. additionally, integrating interdisciplinary insights with algorithmic analysis will likely gain traction for quantifying and mitigating sociodemographic bias. finally, as robust methodologies emerge, we anticipate increased hope for and emphasis on intersectional bias, the overlap of multiple types of bias. ## conclusion we have presented a comprehensive literature survey based on the iterative consideration of 273 works on sociodemographic bias in lms. our proposed typology provides an overview of the current research landscape. we identified promising directions for future research and introduced a 13question checklist designed to guide future research towards more effective and reliable approaches and to avoid pitfalls of previous works. we encourage increased reliance on interdisciplinary methods to better capture and address the nuances of sociodemographic bias in lms.