Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models (2022)

Abstract: Motivations for methods in explainable artificial intelligence (XAI) often include detecting, quantifying and mitigating bias, and contributing to making machine learning models fairer. However, exactly how an XAI method can help in combating biases is often left unspecified. In this paper, we briefly review trends in explainability and fairness in NLP research, identify the current practices in which explainability methods are applied to detect and mitigate bias, and investigate the barriers preventing XAI methods from being used more widely in tackling fairness issues.

## introduction trends in natural language processing (nlp) mirror those in machine learning (ml): breakthroughs in deep neural network architectures, pretraining and fine-tuning methods, and a steady increase in the number of parameters led to impressive performance improvements for a wide variety of nlp tasks. however, these successes have been shadowed by the repeated discoveries that a high accuracy on the held-out test set does not always mean that the model is performing satisfactorily on other important criteria such as fairness, robustness and safety. these discoveries that models are adversarially manipulable @xcite , show biases against underprivileged groups @xcite , and leak sensitive user information @xcite inspired a plethora of declarations on responsible/ethical ai @xcite . two of the common principles espoused in these documents are fairness and transparency. failures in fairness of models is often attributed, among other things, to the lack of transparency of modern ai models. the implicit argument is that, if biased predictions are due to faulty reasoning learned from biased data, then we need transparency in order to detect and understand this faulty reasoning. hence, one approach to solving these problems is to develop methods that can peek inside the black-box, provide insights into the internal workings of the model, and identify whether the model is right for the right reasons. as a result, ensuring the fairness of ai systems is frequently cited as one of the main motivations behind xai research @xcite @xcite . however, it is not always clear how these methods can be applied in order to achieve fairer, less biased models. in this paper, we briefly summarize some xai methods that are common in nlp research, the conceptualization, sources and metrics for unintended biases in nlp models, and some works that apply xai methods to identify or mitigate these biases. our review of the literature in this intersection reveals that applications of xai methods to fairness and bias issues in nlp are surprisingly few, concentrated on a limited number of tasks, and often applied only to a few examples in order to illustrate the particular bias being studied. based on our findings, we discuss some barriers to more widespread and effective application of xai methods for debiasing nlp models, and some research directions to bridge the gap between these two areas. ## explainable natural language processing with the success and widespread adaptation of black-box models for machine learning tasks, increasing research effort has been devoted to developing methods that might give humancomprehensible explanations for the behaviour of these models, helping developers and end-users to understand the reasoning behind the decisions of the model. broadly speaking, explainability methods try to pinpoint the causes of a single prediction, a set of predictions, or all predictions of a model by identifying parts of the input, the model or the training data that have the most influence on the local global self-explaining gradients @xcite integrated gradients @xcite counterfactual lm @xcite ) smoothgrad @xcite deeplift @xcite attention @xcite representer point selection @xcite post-hoc lime @xcite tcav @xcite ) counterfactuals @xcite sear ( ribeiro model outcome. the line dividing xai methods, and methods that are developed more generally for understanding, analysis and evaluation of nlp methods beyond the standard accuracy metrics is not always clear cut. many popular approaches such as probes @xcite , contrast sets @xcite and checklists (ribeiro et al., 2020) share many of their core motivations with xai methods. here, we present some of the most prominent works in xai, and refer the reader to the survey by @xcite for a more extensive overview of the field. we consider a method as an xai method if the authors have framed it as such in the original presentation, and do not include others in our analysis. a common categorization of explainability methods is whether they provide local or global explanations, and whether they are self-explaining or post-hoc @xcite . the first distinction captures whether the explanations are given for individual instances (local) or explain the model behaviour on any input (global). due to the complex nature of the data and the tasks common in nlp, the bulk of the xai methods developed for or applicable to nlp models are local rather than global @xcite . the second distinction is related to how the explanations are generated. in self-explaining methods, the process of generating explanations is integrated into, or at least reliant on the internal structure of the model or the process of computing the model outcome. because of this, self-explaining methods are often specific to the type of the model. on the other hand, post-hoc or model-agnostic methods only assume access to the input-output behaviour of the model, and construct explanations based on how changes to the different components of the prediction pipeline affect the outputs. below, we outline some of the representative explainability methods used in nlp and categorize them along the two dimensions in feature attribution methods, also referred to as feature importance or saliency maps, aim to determine the relative importance of each token in an input text for a given model prediction. the underlying assumption in each of these methods is that the more important a token is for a prediction, the more the output should change when this token is removed or changed. one way to estimate this is through the gradients of the output with respect to each input token as done by @xcite . other methods have been developed to address some of the issues with the original approach such as local consistency @xcite @xcite . rather than estimating the effect of perturbations through gradients, an alternative approach is to perturb the input text directly and observe its effects on the model outcome. two of the most common methods in this class are @xcite and shap (lundberg and lee, 2017). lime generates perturbations by dropping subsets of tokens from the input text, and then fitting a linear classifier on these local perturbations. shap is inspired by shapely values from cooperative game theory, and calculates feature importance as the fair division of a "payoff" from a game where the features cooperate to obtain the given model outcome. allennlp interpret toolkit @xcite provides an implementation for both types of feature attribution methods, gradient based and input perturbation based, for six core nlp tasks, including text classification, masked language modeling, named entity recognition, and others. a third way to obtain feature attribution maps in architectures that use an attention mechanism @xcite is to look at the relative attention scores for each token @xcite . whether this approach provides valid explanations has been subject to heated debate @xcite , however as @xcite notes, the debate has mostly been centered around the use of attention scores as local explanations. there has also been some works that use attention scores for providing global explanations based on the syntactic structures that the model attends to @xcite . extractive rationales (deyoung et al., 2020) are snippets of the input text that trigger the original prediction. they are similar in spirit to feature attribution methods, however in rationales the attribution is usually binary rather than a real-valued score, and continuous subsets of the text are chosen rather than each token being treated individually. rationales can also be obtained from humans as explanations of human annotations rather than the model decisions, and used as an additional signal to guide the model. counterfactual explanations are new instances that are obtained by applying minimal changes to an input instance in order to change the model output. counterfactuals are inspired by notions in causality, and aim to answer the question: "what would need to change for the outcome to be different?" two examples of counterfactual explanations in nlp are polyjuice @xcite and mice @xcite . polyjuice is model agnostic, and consists of a generative model trained on existing, human generated counterfactual data sets. it also allows finer control over the types of counterfactuals by allowing the user to choose which parts of the input to perturb, and how to perturb them with control codes such as "replace" or "negation". mice uses model gradients to iteratively choose and mask the important tokens, and a generative model to change the chosen tokens so that the end prediction is flipped. there are also methods that try to pinpoint which examples in the training data have the most influence on the prediction. the most common approach for this is influence functions @xcite , where the goal is to efficiently estimate how much removing an example from the data set and retraining the model would change the prediction on a particular input. an alternative is representer point selection @xcite , which applies to a more limited set of architectures, and aims to express the logits of an input as a weighted sum of all the training data points. some explainability methods are designed to provide global explanations using higher level, semantic concepts. @xcite use counterfactual language models to provide causal explanations based on high-level concepts. their method contrasts the original model representations with alternative pre-trained representations that are adversarially trained not to capture the chosen highlevel concept, so that the total causal effect of the concept on the classification decisions can be estimated. @xcite adapt testing concept activation vector (tcav) method of @xcite , originally developed for computer vision, to explain the generalization abilities of a hate speech classifier. in their approach, the concepts are defined through a small set of human chosen examples, and the method quantifies how strongly the concept is associated with a given label. finally, some methods produce explanations in the form of rules. one method in this category is anchors @xcite , where the model searches for a set of tokens in a particular input text that predicts the given outcome with high precision. although anchors is a local explainability method in that it gives explanations on individual input instances, the generated explanations are globally applicable. sear @xcite , a global explainability method, finds universal replacement rules that, if applied to an input, adversarially change the prediction while keeping the semantics of the input the same. ## fairness and bias in nlp models unintended biases in nlp is a complex and multifaceted issue that spans various undesirable model behaviours that cause allocational and representational harms to certain demographic groups @xcite . when the demographic group is already marginalized and underprivileged in society, biases in nlp models can further contribute to the marginalization and the unfair allocation of resources. examples include performance disparities between standard and african american english (blodgett and o'connor, 2017), stereotypical associations between gendered pronouns and occupations in coreference resolution @xcite and machine translation @xcite , and false positives in hate speech detection on innocuous tweets mentioning demographic attributes @xcite . in this section, we review some of the most popular methods and metrics to identify such biases. for a more comprehensive coverage, see recent surveys by @xcite and @xcite . most works in ml fairness literature assume that biases in machine learning models originate from misrepresentations in training datasets and merely reflect the societal biases. however, as hooker (2021) explains, design choices can amplify the societal biases, and automated data processing can lead to systematic un-precedented harms. shah et al. (2020) identify five sources for bias in nlp models. selection bias and label bias are biases that originate in the training data. the former refers to biases that are created when choosing which data points to annotate, and includes underrepresentation of some demographic groups as well as misrepresentation due to spurious correlations. the latter refers to biases introduced due to the annotation process, such as when annotators are less familiar with or biased against text generated by certain groups, causing more annotation errors for some groups than others. model bias are biases that are due to model structure, and are responsible for the over-amplification of discrepancies that are observed in training data. semantic bias refers to biases introduced from the pre-trained representations, and include representational harms such as stereotypical associations. finally, bias in research design covers the larger issues of uneven allocation of research efforts across different groups, dialects, languages and geographic areas. research in fair ml has developed a number of metrics to quantify the biases in an ml model. these metrics are usually classified as group fairness metrics and individual fairness metrics @xcite . group fairness metrics focus on quantifying the performance disparity between different demographic groups. some examples are demographic parity, which measures the difference in the positive prediction rates across groups, predictive parity, which measures the difference in precision across groups, and equality of odds, which measures the differences between false positive and false negative rates across groups. individual fairness metrics are based on the idea that the model should behave the same for similar examples re-gardless of the value of a protected attribute. a refinement to this approach is counterfactual fairness, where the criteria for fairness is that the model decision remains the same for a given individual in a counterfactual world where that individual belonged to a different demographic group. in nlp, this notion often appears as counterfactual token fairness @xcite , and is operationalized through test suites that include variations of the same text where some tokens associated with certain social groups are replaced with others, and the bias of the model is measured by the performance disparity between the pairs @xcite . both group fairness metrics and individual fairness metrics are instances of outcome fairness: whether a model is fair is determined solely on the outcomes with respect to various groups, regardless of how the algorithm produced those observed outcomes. foot_0 there is a complementary notion called procedural fairness that is often considered in organizational settings @xcite , which aims to capture whether the processes that were followed to obtain the outcome are fair. in ml, this translates to whether the model's internal reasoning process is fair to different groups or individuals @xcite . for example, outcome fairness for a resume sorting system might be implemented as ensuring that the model has the same acceptance rates or the same precision and recall for groups defined by race, gender, or other demographic attributes. a procedural fairness approach, on the other hand, might aim to ensure that the decision making process of the system only relies on skill-related features, and not features that are strongly associated with demographic attributes, such as names and pronouns. the distinction between procedural and outcome fairness relates to different kinds of discrimination outlined in anti-discrimination laws, namely disparate treatment and disparate impact @xcite . fairness metrics have originally been developed for applications where the social group membership is known, for example in healthcare related tasks. an issue with applying these to nlp tasks is that either the demographic information is not available and needs to be estimated, or some auxiliary signal, such as the mention of a target group or the gender of the pronoun, needs to be used. however, inferring people's social attributes from their data raises important ethical concerns in terms of privacy violations, lack of meaningful consent, and intersectional invisibility @xcite . since determining whether the text is about a certain identity group is easier than whether it is produced by a certain identity group, there are more works investigating the former than the latter. an exception to this is the studies on disparate performance of models on certain dialects such as african american english (aae) @xcite . this is possible due to the existence of a dialect identification tool for aae, which was trained by pairing geo-located tweets with us census data on race @xcite . one source of bias that the nlp community has devoted significant research effort to is word embeddings and pre-trained language models @xcite , which @xcite characterizes as semantic bias. although it is not framed as such, this can be seen as a particular global explanation for biases that the models demonstrate in downstream tasks. however, the effectiveness of these methods has recently been questioned by @xcite who found that there is no correlation between intrinsic bias metrics obtained by embedding association tests, and extrinsic bias metrics on downstream tasks. ## applications of xai in fair nlp to determine the uses of explainability methods in fair nlp, we search the acl anthology for papers that cite the explainability methods listed in section 2, and that include keywords, "fair", "fairness", or "bias". we further exclude the papers that focus on other types of biases such as inductive bias, or bias terms in the description of the architecture. our results show that although there are a number of papers that mention unintended or societal biases as wider motivations to contextualize the work (e.g., by @xcite ), only a handful of them apply explainability methods to uncover or investigate biases. all of the works we identify in this category use feature attribution methods, and except that of @xcite , employ them for demonstration purposes on a few examples. although our methodology excludes works that are published in venues other than acl conferences and workshops, we believe that it gives a good indication of the status of xai in fairness and bias research in nlp. @xcite use shap to demonstrate that adding user features to a hate speech detection model reduces biases that are due to spurious correlations in text, but introduces other biases based on user information. @xcite also apply shap to two example inputs in order to illustrate the political bias of a hate speech model. @xcite aggregate attention scores from bert into global explanations in order to identify which words are most indicative of political bias. some works beyond the papers that our search methodology uncovered on the intersection of fairness for nlp and xai are that of @xcite , which uses sampling and occlusion algorithm of @xcite to detect bias toward identity terms in hate speech classifiers, and that of @xcite , which shows that using human rationales as an additional signal in training hate speech detection models reduces the bias of the model towards target communities. @xcite target individual fairness, and develop a framework to evaluate model bias against particular named entities with a perturbation based analysis. although they do not frame their model as such, the automatically generated perturbations can be categorized as counterfactuals. @xcite suggest the use of two metrics-necessity and sufficiency-as feature attribution scores, and apply their method to uncover different kinds of bias against protected group tokens in hate speech and abusive language detection models. as summarized in there are also some works beyond nlp that use xai to improve fairness of ml models. @xcite , @xcite and @xcite leverage methods from causal inference to both model the causes of the given prediction and provide explanations, and to ensure that protected attributes are not influencing the model decisions through unacceptable causal chains. the disadvantage of these models is that they require an explicit model of the causal relations between features, which is a difficult task for textual data @xcite . @xcite also suggest a causality inspired method that identifies subsets of data responsible for particular biases of the model. @xcite extend shapely values to attribute the overall unfairness of an algorithm to individual input features. the main limitation of all these methods is that they are currently only applicable to low dimensional tabular data. how to extend these methods to explain the unfairness of nlp models remains an open research problem. as abstract frameworks for connecting xai to fair @xcite outline potential synergies between the two research areas. @xcite enumerate different sources of bias, and discuss how xai methods can help identify and mitigate these. ## xai for fair nlp through causality and robustness the framework of causality @xcite is invoked both in fairness and explainability literature. the promise of causality is that it goes beyond correlations, and characterizes the causes behind observations. this is relevant to conceptualizing fairness since, as loftus et al. (2018) argue, there are situations that are intuitively different from a fairness point of view, but that purely observational criteria cannot distinguish. causality tries to capture the notion of causes of an outcome in terms of hypothetical interventions: if something is a true cause of a given outcome, then intervening on this variable will change the outcome. this notion of intervention is useful for both detecting biases and for choosing mitigation strategies. causal interventions are also the fundamental notion behind counterfactual examples in xai. it is easier for humans to identify the cause of a prediction if they are shown minimally different instances that result in opposite predictions. hence, causal explanations can serve as proofs of bias or other undesirable correlations to developers and to end-users. going beyond correlations in data and capturing causal relations is also an effective way to increase robustness and generalization in machine learning models. as @xcite argue, causal correlations are invariant to differing data distributions, while non-causal correlations are much more context and dataset specific. hence, models that can differentiate between the two and rely solely on casual correlations while ignoring the non-causal ones will perform well beyond the strict i.i.d. setting. non-causal, surface level correlations are often referred to as spurious correlations, and a common use case of xai methods for developers is to facilitate the identification of such patterns. a common motivating argument in xai methods for debugging nlp models @xcite , as well as counterfactual data augmentation methods @xcite @xcite , is that unintended biases are due to the model picking up such spurious associations, and xai methods which can be used to improve the robustness of a model against these spurious patterns will also improve the fairness of a model as a side effect. there is indeed evidence that methods for robustness also reduce unintended bias in nlp models @xcite . however, these methods are limited in that they can address unintended biases only insofar as the biases are present and identifiable as token-level spurious correlations. ## challenges and future directions as we saw in sec. 4 and 5, only a few studies to date have attempted to apply explainability techniques in order to uncover biases in nlp systems, to a limited extent. in this section, we discuss some possible reasons for a seeming lack of progress in this area and outline promising directions for future research. local explainability methods rely on the user to identify examples that might reveal bias. one issue in preventing wider adoption of xai methods in fair nlp stems from the local nature of most explanation methods applicable to nlp models. an important step in identifying fairness problems within a model is identifying the data points where these issues might manifest. since local explainability methods give explanations on particular data points, it is left to the user how to pick the instances to examine. this necessitates the user to first decide what biases to search for before employing xai methods, limiting their usefulness for identifying unknown biases. ## conclusion publications in explainable nlp often cite fairness as a motivation for the work, but the exact relationship between the two concepts is typically left unspecified. most current xai methods provide explanations on a local level through post-hoc processing, leaving open questions about how to automatically identify fairness issues in individual explanations, and how to generalize from local explanations to infer systematic model bias. although the two fields of explainability and fairness feel intuitively linked, a review of the literature revealed a surprisingly small amount of work at the intersection. we have discussed some of the conceptual underpinnings shared by both these fields as well as practical challenges to uniting them, and proposed areas for future research.