We Don’t Talk About That: Case Studies on Intersectional Analysis of Social Bias in Large Language Models (2024)

Abstract: Despite concerns that Large Language Models (LLMs) are vectors for reproducing and amplifying social biases such as sexism, transphobia, islamophobia, and racism, there is a lack of work qualitatively analyzing how such patterns of bias are generated by LLMs. We use mixed-methods approaches and apply a feminist, intersectional lens to the problem across two language domains, Swedish and English, by generating narrative texts using LLMs. We find that hegemonic norms are consistently reproduced; dominant identities are often treated as ‘default’; and discussion of identity itself may be considered ‘inappropriate’ by the safety features applied to some LLMs. Due to the differing behaviors of models, depending both on their design and the language they are trained on, we observe that strategies of identifying “bias” must be adapted to individual models and their socio-cultural contexts._Content warning: This research concerns the identification of harms, including stereotyping, denigration, and erasure of minoritized groups. Examples, including transphobic and racist content, are included and discussed._

## introduction the use of large language models (llms) in a wide variety of natural language processing (nlp) tasks and tools, from chatbots to summarization to coreference resolution, is increasing as such models become more widely available both freely and commercially. in such a context, the presence and potential amplification of social biases is of particular concern. we evaluate the presence and implications of representational harms in the output of llms. this is demonstrated for both english (using llama) and swedish (using gpt-sw3). after using the llms to generate stories, we analyze the resulting corpora for representational harms like stereotyping and denigration. we use the equitbl method for distant @xcite and close readings @xcite . our main research question asks, to what extent do llms reflect power asymmetries, including intersectional power asymmetries, in the texts they generate? in particular, we investigate stereotypes, hegemonic norms, erasure of identity in narratives generated by llms. we demonstrate how different methods may be necessary to identify and understand biases across models and socio-linguistic settings, due to divergent behaviours. ## bias statement we consider the overarching concept of 'bias' as the concern for how societal power structures manifest in language technologies. with respect to machine-generated narratives, we locate most of the 'bias' we are concerned with investigating under the umbrella of representational harms, particularly stereotyping and erasure. however, the llms we examine do not always return a narrative text when we prompt them to generate one. thus, we identify several specific system behaviours which we consider distinct harms: ## experimental setup texts generated by llms are presented as the object of study in this paper. to obtain them, we develop a broad set of prompts with the intention of capturing a wide array of stories and circumstances. we vary three elements of the input: the system prompt, the template, and the noun phrase. we then generat texts based on the settings described later in this section. all text generation is performed with the llama-2-70b-chat-hf (70 billion parameters, fine-tuned for performance as a chatbot with rlhf additionally applied) and gpt-sw3-20b-instruct (20 billion parameters, fine-tuned for performance as a chatbot using instruction tuning) models. additional materials are also generated for an exploratory investigation of race in texts produced by gpt-sw3, described in section 5.4. ## methods we use a variety of methods to locate bias in the llms investigated in this paper. some methods, 5 avoiding implications of transphobia and racism. as described in sections 5.1 and 5.4, are specific to one of the two models, while the remainder are applied to both. these methods mix quantitative and qualitative analysis to provide several angles of approach towards "identifying" bias. ## conclusions we find that llms often favor the 'unmarked majority' -if not specified otherwise, names are typical of white us-americans, foot_8 weddings are straight and have a (beautiful) bride dressed in white, etc. gender is also the least likely identity to be 'refused' by llama, as part of its 'safety' features, which may indicate that it is perceived to mark less difference (or constitute less risk) than race or religion. gpt-sw3 does not have this safety feature, and while we can locate more examples of overt racism and sexism, the overall representation is quite similar to the swedish finding described in @xcite . therefore it is notable that the language model did seem to produce more shocking content, including the n-word in both english and swedish, than we might have expected from 'natural' swedish data. however, the nordic pile (which gpt-sw3 is trained on) contains data from flashback @xcite , a large swedish discussion forum with very liberal terms of use, thus also containing liberal amounts of slurs, hate speech, etc. there is also significant amounts of english-language data, which may explain the persistent depiction of us-american stereotypes over culturally swedish ones. together, our findings contribute to the evergrowing scientific consensus that nlp technologies, particularly those based on machine-learning models, replicate and reinforce patterns of bias including stereotyping and erasure. however, it seems that some of the 'safety' measures designed to prevent stereotypes and other behaviours which do not conform to "human preferences" may also contribute to other biases such as erasure by constructing certain groups as 'risky' or 'inappropriate to discuss.' refusal to discuss identity in general on the grounds of "safety" frames identity as taboo, and the user who made the request as inappropriate for even asking. when identity is discussed, the relentlessly positive tone can be alienating, and may in certain applications (for example forum moderation) silence those wishing to find community and talk about their own negative experiences. as language technologies are unavoidably a part of the matrix of domination, the choices made in how they discipline subjects, spread ideas, and facilitate or participate in interpersonal interactions also have unavoidable consequences for society, and their impacts are often more complex than they may seem on the surface.