Discovering and Mitigating Indirect Bias in Attention-Based Model Explanations (2024)

Abstract: As the field of Natural Language Processing (NLP) increasingly adopts transformer-based models, the issue of bias becomes more pronounced. Such bias, manifesting through stereotypes and discriminatory practices, can disadvantage certain groups. Our study focuses on direct and indirect bias in the model explanations, where the model makes predictions relying heavily on identity tokens or associated contexts. We present a novel analysis of bias in model explanation, especially the subtle indirect bias, underlining the limitations of traditional fairness metrics. We first define direct and indirect bias in model explanations, which is complementary to fairness in predictions. We then develop an indirect bias discovery algorithm for quantitatively evaluating indirect bias in transformer models using their in-built self-attention matrix. We also propose an indirect bias mitigation algorithm to ensure fairness in transformer models by leveraging attention explanations. Our evaluation shows the significance of indirect bias and the effectiveness of our indirect bias discovery and mitigation.

## introduction discrimination is the unfair treatment or prejudice directed towards individuals, groups, or certain ideas or beliefs, intentionally or unintentionally. it frequently entails making stereotypes about others and acting in a manner that disadvantages one group while favoring another @xcite . the pervasive nature of bias extends to machine learning, prominently manifesting in the domain of natural language processing (nlp) @xcite . as nlp becomes increasingly integral to everyday life, largely due to the advancements brought by the transformer-based models @xcite , addressing fairness in this field is of utmost importance. in recent years, nlp researchers have undertaken efforts to identify and mitigate discrimina-tion against specific groups, such as gender @xcite , race @xcite , age @xcite , religion @xcite , disability @xcite , etc. they focus on the model's tendency to exploit spurious correlations @xcite between the predicted label and explicit words linked to certain protected attributes, such as "he", "she", "alice", "bob", "russian", "muslim", etc. for instance, in a hate speech detection task, an unfair transformer-based model would see the word "muslim" (also a protected attribute) in a sentence and classify it as hate speech instantly by assigning high attention to the word "muslim", rather than understanding the whole message of the sentence. this is referred to as the legal concept of disparate treatment (supreme court of the united @xcite , that is the outcomes have intended direct discrimination due to choices made explicitly based on membership in a protected class. the existing methods can only handle discriminatory cases where there is a representative token present in the text directly associated with the protected group, e.g., token "muslim" for the islam religion. it also requires the nlp practitioners to manage a pre-determined list of candidate tokens. in contrast to disparate treatment, disparate impact (supreme court of the united @xcite is the legal theory that outcomes should not be different based on individuals' protected class membership, even if the process used to determine that outcome does not explicitly base the decision on that membership but rather on proxy attributes. even without the presence of any direct indicating token in the text, the model still excessively relies on context learned from biased training data, which results in unintended subtle indirect discrimination in the prediction. such indirect association is case by case. it is difficult to pre-determine a candidate token list. remarkably, no prior studies have explicitly delved into indirect discrimination in nlp, to the best of our knowledge. in this work, we want to bridge the gap between disparate treatment and disparate impact in nlp models. the black-box deep learning models tend to over-learn the biased data during training, which results in shortcuts in decision-making without valid explanations. figure 1 illustrates how a model trained to mitigate direct bias against islam religion through "muslim" still falsely categorizes a statement as hate speech because the model's attention is biased emphasized on the sensitive context like the word "quran". an unbiased model would make a negative prediction based on "not always". to investigate bias in the model's local explanations, we first define direct and indirect bias (in section 4). they complement the traditional outcome-association-based group fairness notions, such as demographic parity and equal opportunity. we then propose a novel bias discovery method to evaluate transformer-based models on disparate impact (in section 5). it leverages a secondary transformer-based model dedicated to classifying the protected attribute from the association presented in the training data. we compare the faithful explanations of the primary, potentially biased model, with those of this secondary model. by examining the similarity between their decision-making patterns, we quantify indirect bias through a new proposed metric called the area under the similarity curve (ausc). furthermore, we then proceed to mitigate the detected indirect bias through a similarity-based constraint, which is coupled with mitigating direct bias through data resampling and adversarial learning (in section 6). in our experiment, we show the significance of indirect bias, the effectiveness of our indirect bias discovery and mitigation algorithms, and the advantage of mitigating indirect bias in model explanations (in section 7). thus, our primary contributions are threefold: (1) we establish the problem of fairness in model explanations by formally defining direct and indirect bias; (2) we propose an indirect bias discovery (ibd) framework tailored to quantitatively evaluate indirect bias in transformer models; and (3) we develop a novel indirect bias mitigation (ibm) algorithm that ensures fairness using model explanations. our codes are available at https://github.com/farsheedhaque/indirect-bias 2 related work ## preliminary given an input sequence x with a corresponding protected attribute s and a class label y. x is an ordered sequence of tokens represented as @xmath0. the protected attribute s is the protected group of the person associated with the text. it can be the composer or recipient of the text, or the target whom the text comments on. the value of @xmath1.g., gender is female) is sometimes already expressed in x as a sensitive token s (e.g., "she"), i.e., @xmath2, which is mostly studied by previous works. in this work, we do not require the presence of s in x, where the protected attribute s is a hidden context. the class label y is the prediction target. a text classification model f : @xmath3, y). the model prediction for a sequence x is denoted as ŷ = f (x). specifically, we consider a state-of-the-art transformer-based model. ## indirect bias discovery (ibd) direct and indirect bias evaluate a model's fairness in terms of its decision-making process, a.k.a. model explanations. an unbiased transformerbased model pays high attention to the set of these ground-truth centric tokens t y , whereas a model with indirect bias pays high attention to a set of tokens t s that is associated with s. in practice, either t y or t s is not annotated in the text. a model f can provide local explanations in the form of t y . the key challenge to examine indirect bias is to identify t s . to separate t s from t y and to discover indirect bias in model f we propose an indirect bias discovery (ibd) architecture. figure 2 shows a general overview of our proposed architecture. it is divided into three components -model layer, attention-score aggregation layer, and similarity detection layer. model layer is used to fine-tune our target model f on sequence x. the goal of this fine-tuned f is to successfully predict ŷ where ŷ = f (x). we also get the attention-score matrix a f [{t i } n @xmath4. this layer also has another helper model g fine-tuned to predict the protected attribute s of x such that ŝ = g(x). model g also gives us the attention-score matrix a g [{t i } n @xmath5. then, a f and a g are fed into the next layer as inputs to get the interpretation of the decision-making process of model f and g respectively. attention-score aggregation layer takes highdimensional matrices, a f and a g and maps them into one-dimensional vectors, α f and α g . these vectors encapsulate the importance scores for the token set {t i } n @xmath6, respectively. to achieve this we devised a selfattention score aggregator using different combinations of summation, multiplication, average, and maximum. from different combinations, we took one that performs best on the faithfulness metrics of comprehensiveness and sufficiency @xcite . sufficiency evaluates how sufficient an aggregation is for making a prediction, while comprehensiveness assesses if all the selected elements are essential for the prediction. a minimal reduction in sufficiency and a significant drop in comprehensiveness suggest a high level of faithfulness. our attention-score aggregator follows the operations as in algorithm 1 below. algorithm 1 faithful attention aggregator require: model f , input instance x 1: l, h ← number of layers and heads in f 2: a[l][h] ← attention matrix of layer l, head h in f given x 3: for each combination of head_op, layer_op, token_op in [sum, mul, mean, max] do 4: @xmath7: @xmath8: α = token_op(c) 7: evaluate the faithfulness of α 8: end for 9: return best aggregation combination based on faithfulness metrics and the corresponding α similarity detection layer finds the t y and t s to detect indirect bias in model f . to achieve this, the layer takes α f and α g as inputs. a subset t k f is selected from x, which comprises the top k% importance scores in α f . t k f is a hypothesis of t y based on f . consequently, a subset t k g is selected from x, which comprises the top k% importance scores in α g . t k g is a hypothesis of t s based on g. the similarity between the subsets t k f and t k g is calculated as below. the ausc functions as a quantitative metric for assessing indirect bias present within a given text data denoted as x. this metric primarily targets the identification of indirect bias at the sentence level. nevertheless, the application scope of ausc extends beyond individual sentences, allowing for the calculation of bias across the entire dataset. this process involves taking the ausc values from each sentence and then calculating their average, which gives an overall measure of indirect bias in f w.r.t. the entire dataset. ## indirect bias mitigation (ibm) in this section, we propose a novel indirect bias mitigation (ibm) algorithm to guarantee fairness in model explanations. the goal of our mitigator is to minimize the influence of protected attribute s for a given model f : @xmath9, y). the underlying hypothesis posits that during the training phase, f picks up signals from the context tokens t s associated with the protected attributes s, consequently leading to biased predictions ŷ. to mitigate such indirect bias in model explanations, we design a similaritybased regularization term r to constrain the model to only rely on the key prediction centric tokens t y but not the sensitive context tokens t s . to obtain r, first, we need a pre-trained helper model g : @xmath10. during the training of our f model, we take the attention matrix a f from model f and the attention matrix a g from g model corresponding to the same samples to calculate the cosine similarity between these two matrices using equation 3. ## experiment in this section, we evaluate our proposed indirect bias discovery (ibd) and indirect bias mitigation (ibm) algorithms on sentiment analysis, toxicity detection, and hate speech detection datasets. through case studies, we also demonstrate the significance of indirect bias in model explanations and the advantage of mitigating indirect bias. ## conclusion in this work, we study indirect bias in nlp models, a phenomenon less explored but as significant as direct bias. our contributions include defining direct versus indirect bias, introducing a new framework for quantitatively evaluating indirect bias in transformer models using their in-built selfattention matrix, and proposing a mitigation algorithm to ensure fairness in transformer models by leveraging attention explanations. our evaluation shows the significance and challenging nature of indirect bias in model explanations, and the effectiveness of our proposed discovery and mitigation algorithms. these efforts represent a critical step towards achieving fairness and equity in nlp applications, addressing current research gaps, and guiding future ethical ai development.