Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection (2022)

Abstract: We present a novel feature attribution method for explaining text classifiers, and analyze it in the context of hate speech detection. Although feature attribution models usually provide a single importance score for each token, we instead provide two complementary and theoretically-grounded scores – necessity and sufficiency – resulting in more informative explanations. We propose a transparent method that calculates these values by generating explicit perturbations of the input text, allowing the importance scores themselves to be explainable. We employ our method to explain the predictions of different hate speech detection models on the same set of curated examples from a test suite, and show that different values of necessity and sufficiency for identity terms correspond to different kinds of false positive errors, exposing sources of classifier bias against marginalized groups.

## introduction explainability in ai (xai) is critical in reaching various objectives during a system's development and deployment, including debugging the system, ensuring its fairness, safety and security, and understanding and appealing its decisions by end-users @xcite . a popular class of local explanation techniques is feature attribution methods, where the aim is to provide scores for each feature according to how important that feature is for the classifier decision for a given input. from an intuitive perspective, one issue with feature attribution scores is that it is not always clear how to interpret the assigned importance in operational terms. specifically, saying that a feature is 'important' might translate to two different predictions. the first interpretation is that if an important feature value is changed, then the prediction will change. the second interpretation is that, as long as the feature remains, the prediction will not change. the former interpretation corresponds to the necessity of the feature value, while the latter corresponds to its sufficiency. to further illustrate the difference between necessity and sufficiency, we take an example from hate speech detection. consider the utterance "i hate women". for a perfect model, the token 'women' should have low sufficiency for a positive prediction, since merely the mention of this identity group should not trigger a hateful prediction. however, this token should have fairly high necessity, since a target identity is required for an abusive utterance to count as hate speech (e.g., "i hate oranges" should not be classified as hate speech). in this paper, we develop a method to estimate the necessity and sufficiency of each word in the input, as explanations for a binary text classifier's decisions. model-agnostic feature attribution methods like ours often perturb the input to be explained, obtain the predictions of the model for the perturbed instances, and aggregate the results to make conclusions about which input features are more influential on the model decision. when applying these methods to textual data, it is common to either drop the chosen tokens, or replace them with the mask token for those models that have been trained by fine-tuning a masked language model such as bert @xcite . however, deleting tokens raises the possibility that a large portion of the perturbed examples are not fluent, and lie well outside the data manifold. replacing some tokens with the mask token partially remedies this issue, however it raises others. firstly, the explanation method ceases to be truly model-agnostic. secondly, a masked sentence is in-distribution for the pre-trained model but out-of-distribution for the fine-tuned model, because the learned manifolds deviate from those formed during pre-training in the fine-tuning step. to avoid these problems we use a generative model to replace tokens with most probable n-grams. generating perturbations in this way ensures that the perturbed instances are close to the true data manifold. it also provides an additional layer of transparency to the user, so they can decide whether to trust the explanation by checking how reasonable the perturbed examples seem. although supervised discriminative models rely fundamentally on correlations within the dataset, different models might rely on different correlations more or less depending on model architecture and biases, training methods, and other idiosyncrasies. to capture the distinction between correlations in the data and the direct causes of the prediction, we turn to the notion of interventions from causal inference @xcite . previous work employing causal definitions of necessity and sufficiency for xai have assumed tabular data with binary or numerical features. the situation in nlp is much more complex, since each feature is a word in context, and we have no concept of 'flipping' or 'increasing' feature values (as in binary data and numerical data, respectively). instead, our method generates perturbations of the input text that have high probability of being fluent while minimizing the probability that the generated text will also be a direct cause of the prediction we aim to explain. as our application domain we choose hate speech detection, a prominent nlp task with significant social outcomes @xcite . it has been shown that contemporary hate speech classifiers tend to learn spurious correlations, including those between identity terms and the positive (hate) class, which can result in further discrimination of already marginalized groups @xcite @xcite . we apply our explainability metrics to test classifiers' fairness towards identity-based groups (e.g., women, muslims). we show how necessity and sufficiency metrics calculated for identity terms over hateful sentences can explain the classifier's behaviour on non-hateful statements, highlighting classifiers' tendencies to over-rely on the presence of identity terms or to ignore the characteristics of the object of abuse (e.g., protected identity groups vs. non-human entities). the contributions of this work are as follows: we make the implementation code freely available to researchers to facilitate further advancement of explainability techniques for nlp. foot_0 2 background and related work explanations are often categorized as to whether they are for an individual prediction (local) or for the model reasoning as a whole (global), and whether the explanation generation is a part of the prediction process (self-explaining) or generated through additional post-processing (post-hoc) @xcite . the necessity and sufficiency explanations presented here belong to the class of local explanation methods, as do most of the xai methods applied to nlp data @xcite . it is also a post-hoc method to the degree that it is entirely model-agnostic: all it requires is binary predictions on provided inputs. there are a few classes of popular techniques for explaining natural language processing models (see @xcite for a survey). one approach is feature attribution methods that allocate importance scores to each feature. these can be architecture-specific @xcite , or model-agnostic @xcite . another approach is counterfactual explanations, which provide similar examples to the input in order to show what kinds of small differences affect the prediction of the model @xcite @xcite . these contrastive examples are related to the concept of counterfactual reasoning from the causality literature, that formalizes the question: "would the outcome have happened if this event had not occurred?" in order to determine whether the event was a cause of the observed outcome @xcite . counterfactual explanation methods are often targeted at certain semantic or syntactic phenomena such as negation @xcite or swapping objects and subjects @xcite , and hence do not guarantee that the counterfactuals cover the data distribution around the input text well. in this work, we combine methods from feature attribution and counterfactual generation models. this allows us to calculate scores that capture local feature importance, and provide counterfactual examples as justification for the assigned scores. necessity and sufficiency. these are two notions from causal analysis that capture what one intuitively expects a true cause of an event to exhibit @xcite . several works have recently suggested applying necessity and sufficiency to explain model predictions. @xcite used the actual causality framework of @xcite to calculate necessity and sufficiency scores for tabular data. @xcite suggested an approach to capture the notions of necessity and sufficiency from probabilistic causal models @xcite . @xcite presented a different method for quantifying necessity and sufficiency over subsets of features. we follow the framework of probabilistic causal models, and adopt the definitions from @xcite . in nlp explanations, necessity and sufficiency have been used for evaluating rationales @xcite @xcite foot_1 , however to the best of our knowledge, this is the first work to explore their usage for estimating feature attribution scores. the out-of-distribution problem in feature attribution models. virtually all model-agnostic feature attribution models calculate importance scores by perturbing input features and assign importance according to which feature changes the outcome the most. however, an issue has been raised that these perturbed inputs are no longer drawn from the data distribution that the model would naturally encounter for a given task @xcite @xcite @xcite . this is problematic because then, any change in the model predictions could be caused by the distribution shift rather than the removal of feature values @xcite . recently, @xcite have argued that the problem is due to social misalignment @xcite , where the information communicated by the model differs in non-intuitive ways from the information people expect. one solution to address these issues is to calculate importance scores by marginalizing over counterfactuals that respect the data distribution. @xcite and harbecke and alt (2020) adopted this approach and targeted text data specifically by marginalizing over infills generated by bert. in our preliminary experiments, this resulted in the model putting an overwhelmingly high probability mass to one or few very common words, making the generated perturbations relatively non-diverse. 3 as pham et al. (2021) also pointed out, bert is very good at guessing the masked word, doing so correctly about half of the time. this behaviour results in assigning low importance to highly predictable words regardless of their true importance. for this reason, we choose to use a generative language model to infill masked sections with ngrams. our mask-and-infill approach is similar to that of @xcite and @xcite , who used fine-tuned causal language models to infill masked sections of text with variable length sequences. ross et al. also used the contrasting label to condition the generative model. however, both these works aim to find counterfactual examples as explanations, while we marginalize over them to calculate necessity and sufficiency of each token. ## our method a central idea in causal inference is that of intervention, where a random variable is intervened on and set to a certain value. the intuition is that, if a random variable is the cause of another, then intervening on the first one should affect the other, whereas if they are correlated by other means then the intervention should not have an effect. necessity. let x ← a denote that the random variable x has been intervened so that @xmath0. when talking about a feature vector x, we will denote by x i←a that we intervene on the ith feature value and set it to a. for an input with features x where x @xmath1, the necessity of x @xmath2: sufficiency. the sufficiency of x @xmath3: this means that if x @xmath4, then for inputs in the neighborhood of x that differ in the ith feature value, changing ith feature value to that of a will flip the prediction to f (x) = y. interventions. previous works applying notions of necessity and sufficiency from causal inference to xai assume tabular data. this makes it relatively straightforward to apply these measures to the features since a) it is clear how to assess and compare the ith feature of each input and b) there is little ambiguity in how to change one feature value to another. both these are issues for nlp data, where each feature is a token in the context of the wider text. we argue that the replacements should reflect the likelihood of natural data, but should still be distinct from purely observational correlations in task-specific aspects. to achieve this balance, we sample the replacement values a ′ conditioned both on the other parts of the text and on the opposite class y ′ . if there are two features x @xmath5, the intervention x i←a ′ , where a ′ is sampled in this way results in a ′ being still plausible with respect to the context x @xmath6, but removes the potential indirect effect that x @xmath7, which causes f (x) = y. this allows us to distinguish which of the correlated features the model relies on more for a given prediction. estimation. the formulae for necessity and sufficiency suggest a naive implementation of sampling first from the neighborhood of the input, picking those samples that conform to the conditions, and intervening on the feature of interest and marginalizing over the model predictions to calculate the final value. to perform these steps for each token in a sentence is prohibitively expensive. we therefore perform interventions on subsets of tokens at once, so that one perturbation can be used in the necessity and sufficiency estimation of multiple tokens. we estimate the necessity of a token by perturbing subsets of tokens containing the given token and calculating the average change in model prediction, weighted according to the size of the subset. for calculating necessity, we marginalize over f (c i←a ′ ) where @xmath8,•••j k ←b k for a random subset of features x j 1 , • • • , x j k , not including the original feature i. this means that in our implementation, d n (x) is an interventional distribution around x rather than an observational one. in practice, we estimate a simplified version of this value where we do not explicitly condition on f (c) = y in order to perform the estimation efficiently. we consider the instances where only one or a few tokens are perturbed to have higher probability in d n (x). as such, the weight assigned to a sample with k perturbed tokens is proportional to 1/k. this means that the difference between the original and the perturbed instance is attributed to each perturbed token equally. for estimating sufficiency we take the dual approach. we perturb subsets of tokens excluding the target token, and calculate the difference between the weighted average of the model predictions and the baseline prediction. here too, d s (x) is an interventional distribution where each sample @xmath9,j 1 ←b 1 ,•••j k ←b k for the focus feature x i and a subset of other features x j 1 , • • • , x j k . even though we do not explicitly condition on f (c) = y ′ , d s (x) is biased towards such c because the interventions are conditioned on y ′ . for a sequence of length n, the weight assigned to an instance where k tokens are perturbed is 1/(nk). this means that for an perturbed example that contains only a single token from the original instance, the difference from the baseline will be attributed entirely to that token, whereas if there is k original tokens, the attribution is shared between them. note that d s (x) still assigns a higher probability mass to instances closer to x, but is less peaked than d n (x). figure 1: an illustration of how necessity and sufficiency are calculated for a chosen token "women" in the input "i hate women" that the model classifies as hateful. in the masking step, the subsets of tokens are masked. for the necessity calculation the masked tokens always include the focus word, and for sufficiency they always exclude it. in the infilling step, the generative language model is used to infill the masked sections with n-grams of various lengths. these are then passed to the classifier. the necessity is the proportion of instances where changing the token changes the prediction, and sufficiency is the proportion of instances where changing other tokens does not change the original prediction. the infills are real examples generated by our method. ## explaining hate speech models for tasks with very skewed class distributions such as those for binary hate speech classification, it is not intuitive to ask for explanations for the majority class predictions: it is difficult to answer why a regular utterance such as "i would like some coffee." is not hate speech. this echoes the argument of @xcite that humans demand explanations only for selective and surprising aspects of an occurrence. we assume that there exists a majority, 'neutral' class, and aim to provide explanations only for the minority, 'positive' class. at the core of our approach is sampling replacements x i ← a ′ that are interventional with respect to the task, but also have high likelihood in the context of other tokens. we implement such a perturbation model for explaining positive predictions of hate speech classifiers, and explore what information necessity and sufficiency provide for this task through quantitative and qualitative analysis. our implementation is task specific, since we sample from the data labelled as 'neutral' only; however, it is not dataset specific in that it allows comparing hate speech classifiers that are trained on different datasets. figure 1 illustrates the calculations of necessity and sufficiency for an example classifier. ## experiments for our experiments, we focus on investigating the necessity and sufficiency of identity mentions for a sentence to be detected as hate speech, since a known bias in many hate speech detection models is that mere mentions of identity terms result in false positive predictions @xcite . for a set of instances that contain mentions of identity terms we leverage hatecheck @xcite , which is a suite of functional tests targeted at diagnosing weaknesses of hate speech classifiers. tests are constructed from hand-crafted templates, where the target is picked from a predefined set of protected groups. to evaluate our explanation method, we train six bert classifiers on three different datasets @xcite @xcite , and obtain the necessity and sufficiency of the identity terms on explicitly hateful test cases in hatecheck that target women and muslims. details for the datasets and classifiers can be found in appendix b. we train classifiers on both 'hate speech' labels and on more general 'abusive' language labels in order to observe the effects of this distinction on the necessity and sufficiency values for identity terms. the difference between the two is that abusive language does not need to target a protected group identity. accordingly, our first hypothesis is: hypothesis 1: we will observe lower necessity for the identity terms for those models that are trained on the 'abuse' labels compared to the ones that are trained on the 'hate' labels. we further hypothesize that different necessity and sufficiency scores for identity mentions in explicitly hateful test cases indicate different biases, and correlate with how well a model does in the hate-check functionalities that aim to capture those biases (see these women disgust me so much. these muslims disgust me so much. hypothesis 2: if a model has high sufficiency scores for identity terms in explicit expressions of hate (functionalities f1, f2 and f3), this should indicate that the model is over-sensitive to identity terms. therefore, we expect it have increased error rate in f18 and f19, which consist of neutral or positive statements that mention identity terms. hypothesis 3: if the necessity scores for identity terms are low in explicit expressions of hate, we can conclude under-reliance on the identity terms, and over-reliance on the context. consequently, we expect that the model will perform poorly on f22, f23 and f24, which capture abuse not targeted at protected identity groups. ## results and discussion an example necessity and sufficiency attribution is given in figure 2 . it shows that for this input, the token 'muslims' is more sufficient compared to 'women', and the token 'disgust' is more necessary in the context of 'women' than that of 'muslims'. according to our first hypothesis, we expect the models that were trained on the abuse versions of each dataset to have lower necessity for identity terms compared to those that have been trained on hate labels. indeed, in the results also clearly support our second hypothesis that if an identity mention has high sufficiency on explicit examples for a given model, then this model is over-sensitive to the identity term. comparing the sufficiency of women and muslims in and vidgen2021-abuse display the highest sufficiency for women, and correspondingly have the highest error rates on these test cases for women. davidson2017-abuse has the lowest sufficiency for muslims, and the lowest error rate for this target. our third hypothesis is that low necessity for identity terms will be correlated with positive predictions for abusive instances that do not target a protected identity. in ## conclusion this work is a step towards more informative and transparent feature attribution metrics for explaining text classifiers. we argue that standard token importance metrics can be ambiguous in terms of what 'importance' means. instead, we adapt the theoretically-grounded concepts of necessity and sufficiency to explain text classifiers. besides being more informative, the process of generating these two metrics is intuitive and can be explained to lay people in terms of "how much the perturbations in input change the output of the classifier". moreover, the input perturbations can be presented to the users, leading to a transparent and understandable explainability framework. considering the complexities of perturbing textual features, we introduced a practical implementation to compute the necessity and sufficiency of the input tokens. taking hate speech detection as an example application, we showed that sufficiency and necessity can be used to explain the expected differences between a classifier that is intended to detect identity-based hate speech and those trained for detecting general abuse. we also leveraged these metrics to explain the observed over-sensitivity and under-sensitivity to mentions of target groups, issues that are tightly related to fairness in hate speech detection. while the current work focused on binary hate speech detection for english-language social media posts, in future work, we will explore the effectiveness of these metrics in generating explanations for other applications and languages. we will also explore how the new metrics can improve the debugging of the models or communicating the model's decisionmaking process to the end-users.