Power(ful) Associations: Rethinking “Stereotype” for NLP (2025)

Abstract: The tendency for Natural Language Processing (NLP) technologies to reproduce stereotypical associations, such as associating Black people with criminality or women with care professions, is a site of major concern and, therefore, much study. Stereotyping is a powerful tool of oppression, but the social and linguistic mechanisms behind it are largely ignored in the NLP field. Thus, we fail to effectively challenge stereotypes and the power asymmetries they reinforce. This opinion paper problematizes several common aspects of current work addressing stereotyping in NLP, and offers practicable suggestions for potential forward directions.

## introduction in the last decade, research into "bias" in natural language processing (nlp) has been increasing at a dramatic rate @xcite . this body of work seeks to identify and mitigate social and material harms perpetuated by nlp systems due to historical patterns of oppression. however, this work often fails to ground itself in theory about the mechanisms of harms and their contextual nature @xcite . i argue that bias mitigation in general, and stereotype mitigation in particular, can never be completely successful in "the general case" and likely will only ever partly succeed for purpose-built systems. nlp technologies are parts of complex sociotechnical systems, and interact with our wider social world as actors in systems of power and oppression. although a perfect system will remain out of reach, we can and should continue to seek improvement and reduce harmful aspects of our flawed systems. stereotypes, and their "counters," are moving targets that change over cultural settings and over time. additionally, the social groups targeted by stereotypes are not monolithic, and members will experience stereotypes and their harms differently from each other as well as hold different opinions about how to be "better represented" by language technologies and their outputs. addressing these factors requires attention to intersectional power dynamics, awareness of the cultural and sociolinguistic context of nlp technologies, and clarity around the normative judgements annotators must make @xcite . i explore the gaps between a cultural media studies informed approach to "stereotype" and the more prototypical ways of conceptualizing and operationalizing nlp approaches found in the literature, following a few well-known exemplars to illustrate these trends. i identify several places (defining a "bias" boundary line; the idea of "anti-stereotype"; universalizing; and a reliance on metrics) where such gaps likely impede our ability as nlp practitioners to actually minimize harm. in the final sections, i provide and amplify several suggestions to ways we can change our practices communally and individually to better handle stereotyping in the future. ## related work stereotyping is commonly understood in the context of nlp research as the strong association between a social group and stereotypical attributes such as descriptors or occupations @xcite . operationalization is threatened by a lack of clear definitions of 'steroetype' or agreement on desired model behaviors @xcite . typically missing is a deeper and theoretically grounded understanding of how stereotyping enacts harm, which is necessary to counter such harms in nlp settings. because stereotypes are transmitted and maintained linguistically @xcite , and because language has material effects @xcite , it is important to be deliberate in how we address them in language technologies. somewhat circularly, stereotyping is both a form of bias and a type of harm, i.e. a quality which defines a system behaviour to be "biased". stereotypes are implicated in both allocational harms via attribution of downstream behaviors, and representational harms per se @xcite . datasets for identifying (challenge sets) and reducing (training sets intended for fine-tuning) stereotypical associations in nlp systems have been produced for both english and multi-lingual settings. examples include crows pairs @xcite , seegull @xcite , the multilingual racial hoaxes corpus @xcite , and stereoset @xcite . in addition, there are a variety of other bias identification and mitigation methods that use "stereotypical associations" as their definition of bias, such as the word embedding association test @xcite and pronoun resolution challenge sets like winogender @xcite or winobias @xcite . . works addressing stereotyping in nlp persist in treating stereotype as a discrete, often binary, categorical attribute. associations are either stereotypical (implicitly: harmful) or they are not (implicitly: unobjectionable). despite some acknowledgment of the fact that stereotypes and stereotyping's harm may depend on many contexts such as culture, language, and in-vs out-group status, this discrete definition remains quantitative and reliant on an-notators whose positionality may not be reported @xcite . this in turn makes it difficult to establish the context in which annotator judgments about "stereotype" are made, and thus both their accuracy (out-group annotators may miss stereotypes) and whether they may be applied to other contexts (e.g. cross-culturally). there are exceptions, such as fraser et al. (2021) who use the stereotype content model (scm) of stereotyping to identify "anti-stereotypes" in a more nuanced way. the scm asserts that there are two orthogonal dimensions, warmth (perceived intent to help, vs. harm; (dis)like) and competence (perceived ability to act on this intent; (dis)respect), which all stereotypes form around @xcite , and that different combinations are associated with distinct emotional reactions to the stereotyped. however, as the name implies, the scm focuses on the content, i.e. the association between group and quality or behavior, of the stereotype. this fails to account for the form or narrative of stereotype, which comprises mechanisms of stereotype transmission and the ways in which stereotypes play into our individual and collective sense-making. we miss the power relations: which qualities of warmth and competence are valued, by whom, applied to whom, in which contexts? much like identifying hate speech @xcite or misinformation @xcite , the "facts" of an association alone are insufficient to robustly identify that association as stereotypical. instead, we must turn to theories which allow us to take into account more of the context that surrounds this content to enable normative judgment. ## theoretical grounding stereotyping is culturally embedded, and as such its harms are context-dependent. all associations between groupings and attributes or qualities are cultural, but which are "harmful" is harder to determine. we turn to cultural media studies, which is better-equipped to handle texts and narratives, to delineate between type and stereotype in nlp. ## problematizing "anti"-stereotype stereotyping's harms follow from its function as a reductive essentializer, denying full personhood to members of such groups. attempting to counter stereotypes, therefore, must be done in ways that are not also reductive or essentializing, and which avoid the trap of 'countering' surface stereotypes with "deep structure" stereotypes. stereoset @xcite has been critiqued extensively by @xcite , but is still in wide use and offers a convenient example for deconstructing this tension in the form of a mask-filling task: girls tend to be more than boys. soft stereotypical determined anti-stereotypical the surface stereotype, that women are soft, is identified as stereotypical. it essentializes women as weak, and in doing so supports clear and established hegemonic power structures. it works to divide groups (men are put in opposition to women ); to define lines between "normal" (soft women, hard men) and "deviant" (hard women, soft men) members of these groups; and to secure hierarchy (@xmath0, @xmath1. thus, it conforms to the heterosexual matrix as "oppositionally and hierarchically defined" @xcite . however, in labeling "determined" girls as antistereotypical, the example fails to recognize the deep stereotype that women are (or ought to be) girl-bosses, held to incredibly high standards of perfection. the underlying fantasy (that women are more capable than men in a valued dimension) threatens patriarchy; framing women as "determined" plays into this while implying that girls might 'need' to be determined as they lack the natural capacity of boys -thus working to reduce the threat. the resulting tension traps women at both levels of stereotype, pressuring them to be both soft caregivers and determined girl-bosses without being 'too much' of either, an impossible task. both roles emphasize positivity @xcite and work women complexity. furthermore, judgements of this type often lack an intersectional lens: not all individuals in a group experience the same stereotypes in the same way (see, e.g. @xcite @xcite ). the stereotype of women being "soft" is racialized. it is typically applied to white (and east asian) woman, but not women of color -particularly black women, who are instead characterized as "strong", angry, or violent @xcite . latina women may be caught between both stereotypes: traditional, domestic "good girls" and loud, criminal, sexualized "bad girls" @xcite . while white (cis, straight, perisex) women are essentialized as delicate, infantilized creatures who require protection, their bipoc (trans, queer, intersex) sisters are instead denied the quality of softness, and through it femininity. as a deep stereo-type, "determined" also serves to trap womenworking class women, single mothers, immigrant women, among many -in narratives of struggle that deny their fully-realized personhood. this stereotype thus also serves as a tool of white supremacy. women who do not fit the mold are, the violent logic dictates, deviant or not real women. through this characterization, their personhood is denied. the softness of white women is also weaponized against black men and other minoritized groups, when positioned as victims to enable persecution for imagined offenses (see, e.g. phipps ( 2021 )).the harm goes deeper than the surface. the concept of "anti-stereotype" is thus quite complicated, and its identification is a moving target. @xcite show that annotators tasked with selecting anti-stereotypes are inconsistent in how they conceptualize and operationalize this binary, and as we have just demonstrated "antistereotypical" associations may still be oppressive. ## one size fits all? other complicating factors for identifying and mitigating stereotype are disentangling "stereotypes" from "associations," and recognizing that this is not always possible if the difference is only a looselydefined "harm." stereotypes may be globally harmful (reinforcing power asymmetries) and still compelling or empowering locally, at a personal level @xcite . this trouble is not unique to stereotypes: the utility and morality of slur reclamation is often a matter of considerable debate within minoritized groups. these surround who can legitimately use the "reclaimed" term, under which circumstances, for the usage to be reclamatory while also accomplishing the goals of reclamation @xcite . such nuances are an issue in toxicity detection, where systems designed to prevent abuse of a group instead push them out @xcite . ## troubling metrics it is well known within nlp that how we operationalize bias, and therefore how we implement interventions designed to counter it, has consequences which may include obfuscating biases @xcite . this is perhaps most famously shown by @xcite , who demonstrate that debiasing methods for word-embeddings do not remove those biases, only hide them. more recently, @xcite demonstrate that even when large language models are fine-tuned to avoid making overtly racist associations, their output still demonstrates concerning covert (or implicit) racism; and that these outputs directly result in downstream allocative harms such as disproportionate rates of conviction and harsher sentencing. furthermore, how we operationalize groupings needs to constantly be re-interrogated. as a field, we risk entrenching particular categories by repeatedly reaching for the same ones -nearly half of the past decade of papers investigating bias in nlp focus on (binary) gender @xcite . ## call(s) to action this is not the first paper to voice specific calls to nlp researchers and practitioners concerned with bias and injustice in their field. we must treat representational harms as harms per se (blodgett et al., 2020), leverage feminist theories and research strategies @xcite ; and address the specific needs of minoritized groups @xcite . ## conclusion addressing the matter of "stereotype" in nlp requires a solid theoretical grounding to avoid inadvertently introducing or reproducing other harms. failure to engage with this theory produces sites where the gap impedes our ability as a field to truly mitigate harm: drawing lines of what is and is not "acceptable" associations; failing to address both surface and deep structures of stereotype; universalizing without attention to context; and categorization. some shifts towards more grounded ways of working with stereotype in nlp may be individual, while others likely require infrastructural support.