Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention (2022)

Abstract: Natural Language Processing (NLP) models are found to exhibit discriminatory stereotypes across many social constructs, e.g. gender and race. In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic damage. Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find that stereotypes are also encoded in the attention layer. Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity. Our experiments confirm that reducing bias from attention effectively mitigates it from the model’s text representations.

## introduction natural language processing (nlp) is increasingly penetrating real-world processes such as recruitment @xcite , legal systems @xcite , healthcare @xcite and web search @xcite . part of this success is attributed to the underlying embedding layer which encodes sophisticated semantic representations of language @xcite . however, this wide adoption has called into question the fairness aspect of modern text encoders. recent research exposed them for replicating discriminatory social biases which may cause unintended and undesired model behaviors with respect to social groups @xcite @xcite . as an example, a resume filtering system built on top of such biased encoders display overly improper preferences towards male applicants @xcite . methods to debias text encoders have recently been proposed, ranging from counterfactual data augmentation (cda) @xcite , projection on bias-free subspaces @xcite , contrastive learning @xcite , adversarial attacks @xcite or simply extending existing debiasing techniques from static word embeddings @xcite @xcite . however, these methods have shown mixed results, often failing to reduce the amount of bias to a satisfactory degree @xcite @xcite . we argue that part of this shortcoming owes to the fact that current debiasing techniques, by operating exclusively on the model's embeddings, are not removing bias entirely. in this paper, we propose that some biases can also be encoded in the attention mechanism, and these stay relatively out of reach for methods that do not manipulate attention directly. to illustrate how biases are reflected in attention, we show some attention heads of bert @xcite in figure 1 foot_0 . consider the following sentence "the doctor asked the nurse a question." aiming to analyze how every word representation relates to different demographics, we add a dummy second input consisting of words representing distinct groups (e.g. he and she after the [sep] token 2 ). figure 1( @xcite illustrates that doctor pays much more attention to he than to she 3 , while figure 1(b ) reveals that nurse attends to she. this finding suggests that gender stereotypes are encoded in attention weights. likewise, in figure 1 (c), math is more related to asian than to white or black, the debiasing method proposed by @xcite conforming to the famous racial stereotype casting asians as good mathematicians @xcite . more intriguing is figure 1(d ), where we find that even after applying the bias reduction method proposed by @xcite , gender bias is still reflected in the attention of the supposedly debiased text encoder. we experimented with other debiasing methods and found similar results. these examples convey that biases can be hidden in the attention mechanism, and thus pose the risk of being recovered in representations and predictions. in this paper, we propose a novel bias measure based on attention weights in order to quantify the amount of bias encoded in attention heads. we show that modern text encoders display substantial amounts of bias in their attention components. also, we quantitatively show that current debiasing methods do very little to mitigate social stereotypes, and merely conceal them in the attention layer. then, we propose attention-debiasing (attend), an attention-based debiasing approach which works as follows: given that attention weights conform with undesired biases (e.g., doctor attending to he, and nurse to she in figure 1 ), we finetune the parameters of the text encoder of interest such that it learns to produce equal attention scores for every word in the input sentence with respect to social groups. returning to the example of figure 1 , at-tend redistributes attention scores of doctor such that it attends to he and she with the same intensity, thus eliminating any preference toward one of the groups. however, alterations to the attention of doctor on the remaining words of the input sentence must be kept to a minimum in order not to corrupt the semantic understanding of the original text encoder. to do that, we distill the original attentions from an unaltered teacher text encoder @xcite . in this setting, we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset. we suggest that by equalizing attention weights, text encoders forget biased associations between groups and attributes. thus, for a given input sentence, whether it mentions a man or a woman, a muslim or a christian, the model's attention on the mentioned group is the same, which leads to very close text representations, and hence identical predictions. in our experiments, we address gender, racial and religious biases, and show that attend not only reduces social stereotypes from the attention layer itself, but also from representations and predictions when applied to the tasks of textual inference and hate speech detection. we release our code and data on github. foot_2 2 related work ## bias quantification using attention despite the applicability of our work on any text encoder that is built upon self-attention, we focus in this paper on models based on the encoder side of the transformer architecture, such as bert @xcite , roberta @xcite , or albert @xcite . this owes to the decoder side being usually used in auto-regression tasks, and less often to encode text. in this section, we present our metric to compute the amount of bias that is encoded in attention across an entire corpus s. first, we identify bias types of interest such as gender, race, religion. this is achieved by defining a set of tuples g for every religion gender muslim, christian, jewish he, she quran, bible, torah man, woman we observe that bert's attention heads encode different stereotypes with different intensities, conforming to the findings of @xcite . also, the lower layers of attention appear to encode more bias than the top layers since their heads are much darker. we believe this to be the consequence of lower layers being more aware of the input tokens, while top layers are fed transformations of the input as it flows through the attention stack. we also compute attention bias on bert heads after applying different existing debiasing approaches, and present the results for gender bias in figure 3 . when we compare the heatmaps in figure 3 to the heatmap of figure 2 (a), we notice that they are very similar, and attention bias is hardly removed. in some cases, it is even amplified (e.g. head 3 in layer 10 in figure 3(a) ). we stipulate that even though those debiasing methods show acceptable results with embedding-based bias evaluations, they appear to ignore the bias reflected in attention and thus can be recovered at prediction. in the next section, we present our own debiasing methods that aims to reduce bias from the top k most biased attention heads. ## debiasing method the first step of attend is augmenting each sentence s in the training corpus s with an artificial second input s g consisting of words related to groups of a given bias type, as explained in section 3 and in the examples of figure 1 . then, we finetune the encoder's parameters such that the top k most biased heads produce equalized attentions on groups, i.e. each token in s pays the same amount of attention to tokens of s g , thus eliminating preferences and stereotypes. we minimize semantic loss by compelling the model to learn the original semantics from an unaltered teacher model by copying its internal attention. we schematize the operation of attend in fig- (a) (b) (c) figure 3: attention bias in bert base broken out by layer and head after the application of several debiasing methods to reduce gender bias. these debiasing methods are (a) cda (b) sent-d (liang et al., 2020a), (c) kaneko kaneko and bollegala (2021) (a) (b) figure 4: overview of an attention head before (a) and after (b) debiasing and (4) attentions of s g on s g . debiasing consists in making the columns of block 2 equal. in other words, each token in s pays the same amount of attention to all the groups as indicated in the right side of figure 4. we preserve the semantics of the original text encoder by keeping block 1 of figure 4 unchanged. both blocks 3 and 4 are irrelevant to the results, since they denote attentions of our artificially inserted second input s g . so, we do not impose any restrictions on them. in the following, we describe the important details of attend. ## evaluation in this section, we first describe our experimental setup, then evaluate attend from two viewpoints: fairness and representativeness. fairness is traditionally evaluated with two types of metrics: intrinsic metrics that measure bias in text representations regardless of their application, and extrinsic metrics that quantify bias in downstream tasks that text representations enable. we acknowledge that intrinsic metrics have recently been criticized @xcite @xcite . however, we believe that a strong evaluation of bias should include both intrinsic, extrinsic and qualitative methods to draw a comprehensive evaluation. since @xcite surmise that stereoset and crows-pairs are more stable than other intrinsic measures of bias (e.g. weat @xcite or seat ( may et al., 2019) ), we use them in this work. for extrinsic metrics, we evaluate our method on the tasks of textual inference and hate speech detection. due to space limitations, we move the second one to appendix a.3, in addition to other qualitative evaluations and ablation studies foot_6 . ## conclusion we proposed in this paper to pay closer attention to the attention mechanism of text encoders. specifically, we find that social bias also resides in attention weights, in addition to in representations and embeddings. we characterize attention bias in text encoders by looking at which demographics are most relevant given a stereotypical scenario under different attention maps in various layers. our bias quantification method is a weighted average of pearson correlations between attention allocations for demographic group words. we also propose a novel debiasing method by modifying the self-attention weights so as to ensure equal attention activations across all group words for each token in each input sentence. at the same time, we use knowledge distillation from a teacher text encoder to preserve the useful semantics contained within. finally, we utilize negative sampling with non-demographic word sets as the second sentence, where the teacher objective rather than attention equalization objective is applied, to prevent sentence-pair functionality in text encoders from being destroyed. we find that by mitigating biases from attention, the overall model bias is also reduced. we demonstrate this with various experiments that probe for bias internally, and when text encoders are used in downstream tasks, namely sentence inference and hate speech detection with limited costs to semantic usefulness.