Layered Bias: Interpreting Bias in Pretrained Large Language Models (2023)

Abstract: Large language models (LLMs) like GPT and PALM have excelled in numerous natural language processing (NLP) tasks such as text generation, question answering, and translation. However, they are also found to have inherent social biases. To address this, recent studies have proposed debiasing techniques like iterative nullspace projection (INLP) and Counterfactual Data Augmentation (CDA). Additionally, there’s growing interest in understanding the intricacies of these models. Some researchers focus on individual neural units, while others examine specific layers. In our study, we benchmark newly released models, assess the impact of debiasing methods, and investigate how biases are linked to different transformer layers using a method called Logit Lens. Specifically, we evaluate three modern LLMs: OPT, LLaMA, and LLaMA2, and their debiased versions. Our experiments are based on two popular bias evaluation datasets, StereoSet and CrowS-Pairs, and we perform a layer-by-layer analysis using the Logit Lens.

## introduction motivation: large language models (llms) have risen to prominence, revolutionizing the field of natural language processing (nlp). these models, such as opt @xcite and llama @xcite , are trained on vast and diverse data sources encompassing webpages, wikipedia, books, scientific papers, and other online content. while this broad spectrum of data ensures a rich representation of the world's knowledge, it also serves as a double-edged sword. on one side, it represents a democratic and diverse range of ideas, yet on the flip side, it exposes the models to inherent social biases. in recent years, the nlp community has prioritized studying biases in llms. early work by @xcite revealed gender and ethnic biases in word embeddings like word2vec and glove. this trend of identifying biases continued with more complex models like bert, where researchers examined how biases are encoded and propagated @xcite . researchers have also developed datasets, such as stereoset @xcite and crows-pairs @xcite , specifically to measure and understand these biases. @xcite delved into the effects of biased data, especially from human annotators, on the behavior of models. alongside identification, efforts have been geared towards the mitigation of bias in llms. techniques such as iterative nullspace projection (inlp) @xcite and counterfactual data augmentation (cda) @xcite have been proposed and implemented to mitigate biases in llms. nevertheless, many of the existing studies have examined and evaluated biases in llms in a more coarse-grained manner, and it is unclear how the debiasing techniques affected the llms in deeper neural layers. we aim to address this research gap by conducting an in-depth analysis to interpret layer-wise bias in pretrained llms. interpretability in llms has gained significant attention due to the implications of understanding and explaining model decisions. prior research has leveraged techniques such as attention visualization @xcite , @xcite , and shap @xcite to uncover feature significance. integrated gradients, introduced by @xcite , offers insights into how deep learning models relate predictions to input features, thereby illuminating their decision paths. another groundbreaking tool is the logit lens by nostalgebraist (2020). it reveals that when the hidden states of each gpt-2 layer @xcite are decoded with the unembedding matrix, the ensuing distributions consistently narrow down, leading to the model's final output. this approach has paved the way for recent research, with studies employing logit lens to interpret transformer weight matrices @xcite @xcite . building on these foundations, we adapt logit lens in our pursuit to unravel biases across the layers of pretrained llms. ## experimental setup in this section, we outline the crucial elements of our experimental analysis. we begin by discussing the benchmark datasets. this is followed by an overview of the llms used in our experiments. lastly, we delve into the debiasing and interpretability techniques applied to the llms. ## bias analyses research on bias has largely examined the overall tendencies of models to display biased behavior. given that humans can manifest bias in myriad linguistic expressions, and llms are becoming increasingly proficient at replicating human language, the benchmarks used in these studies might not capture the full spectrum of bias due to their fixed sentence structures. with this in mind, we aim for a more nuanced understanding of bias in these models. our study seeks to address two primary questions: we employ the logit lens to explore how bias develops across the various layers of our models. figure 1 presents the ss values across layers using the stereoset dataset. similar visualizations for crows-pairs and ogb datasets can be seen in figure 3 and figure 2 , respectively. our observations indicate that the variation of ss across layers is more consistent in stereoset than in the other datasets. specifically, in crows-pairs, there's a noticeable undulation in values, but overall, they tend furthermore, the opt_6.7b model initially demonstrates an anti-stereotypical stance on religious bias in its early layers. however, this stance rapidly shifts to a more stereotypical one in subsequent layers. in the ogb dataset, visualized in figure 2 , the discrepancy between pretrained and debiased models is more pronounced for llama_7b and llama-2_7b. we've further bifurcated our analysis based on male and female-dominated jobs, examining how each model's preference for genderspecific words changes layer by layer. to delve deeper, we conducted a manual analysis of layer-wise bias using a sample from the crows-pairs dataset. we fed both stereotypical and antistereotypical sentences to the pretrained llms and documented the predictions at each layer. this process was then repeated for the debiased llms. as an illustration, figure 4 showcases results from the llama_7b model for the stereotype sentence, "my mom spent all day cooking for thanksgiving." an intriguing observation in our analysis is the prediction of probable bi-gram tokens by both the initial and, in certain instances, later layers. for instance, in figure 4 (a), the terms "light" and "long" emerge as predictions for some layers at the "day" token step. a comprehensive list of examples can be found in our appendix section. ## discussion and conclusion since the advent of llms, numerous studies have aimed to decode their operations, exploring questions like where they store factual information, how they learn from context, and more recently, their safety. however, while many investigations have delved into the outputs of these models, few have examined the evolution of their behavior within the neuron layers. in our research, we delve into the individual layers of llms to understand their potential for bias. we assess several current models layer-by-layer using widely recognized datasets. through a detailed manual analysis of token predictions in intermediate layers, we elucidate the effects of debiasing measures. our results reveal that different layers in llms behave uniquely concerning various biases, with each model presenting its own pattern. moreover, every dataset paints a distinct picture; for example, the ogb dataset exhibits a marked bias for male terms in male-dominated professions, contrasting sharply with the near-neutral gender bias in the crows-pairs dataset. variant text prompt my friend is a janitor, and just like most of them, he's pretty pretrained my friend is a janitor, and just like most of them, he's pretty icky. debiased my friend is a janitor, and just like most of them, he's pretty uneducated.