Concept-Based Explanations to Test for False Causal Relationships Learned by Abusive Language Classifiers (2023)

Abstract: Classifiers tend to learn a false causal relationship between an over-represented concept and a label, which can result in over-reliance on the concept and compromised classification accuracy. It is imperative to have methods in place that can compare different models and identify over-reliances on specific concepts. We consider three well-known abusive language classifiers trained on large English datasets and focus on the concept of negative emotions, which is an important signal but should not be learned as a sufficient feature for the label of abuse. Motivated by the definition of global sufficiency, we first examine the unwanted dependencies learned by the classifiers by assessing their accuracy on a challenge set across all decision thresholds. Further, recognizing that a challenge set might not always be available, we introduce concept-based explanation metrics to assess the influence of the concept on the labels. These explanations allow us to compare classifiers regarding the degree of false global sufficiency they have learned between a concept and a label.

## introduction in various natural language classification tasks, particularly in abusive language detection, certain concepts are known to be strong signals for the label of interest. these concepts are often overrepresented in the respective class of the training set, making them susceptible to being learned as potential causes for the label. consequently, the classifier over-relies on these concepts and ignores the broader context, leading to reduced generalizability @xcite . hence, to ensure models are robust and reliable, it is crucial to develop methods that can detect these over-reliances in various natural language classification tasks. in the context of abusive language detection, we consider the concept of negative emotions. the figure 1 : probability of offensiveness generated by the tweeteval classifier @xcite . the classifier has learned a false global sufficiency between negative emotions and the label of offense. it over-relies on this concept and ignores the broader context. we explore a scenario where a user, aware of the importance of negative emotions for their use case, wants to evaluate and compare a set of trained models. their goal is to identify and eliminate those models that are more prone to generating inaccurate results due to an overemphasis on negative emotions as primary indicators. for that, we use concept-based explanations to test if a model has learned a false global causal relationship between a user-identified concept and a label where the true relationship is a correlation. note that global causal relationships explain the model's output across an entire dataset, as opposed to local causal explanations, which concern the dependency of an individual prediction on a specific input feature. concept-based explanations are a class of explainability methods that provide global explanations at the level of human-understandable concepts @xcite . while local explanations help the users understand the model's reasoning for an individual decision with respect to the input features, global explanations are critical in comparing the processes learned by models and selecting the one that best suits the needs of a use case @xcite . global explanations might be obtained at the level of input features through aggregating local explanations @xcite . alternatively, global-bydesign methods (e.g., probing classifiers @xcite ) can be used to gain insights at higher levels of abstractions, such as linguistic properties or human-defined concepts. 2 2 similar to most feature importance explainability methods (e.g, @xcite ; lundberg and lee (2017a)), concept-based explanations are originally designed to measure the importance of a concept. the intuitive meaning of importance usually refers to correlation, and it can be interpreted differently based on two notions of causality: necessity and sufficiency @xcite . local explainability methods usually focus on features that are of high local necessity or high local sufficiency for the label @xcite @xcite , thus considered important by human users. however, at the global level, all features must be interpreted in a larger context for accurate decision-making. we aim to determine if concept-based explanations can be utilized to evaluate whether a trained binary classifier for abusive language detection has learned a false global sufficiency relationship between the label and the concept of negative emotion. our code and data are available at https://github.com/isa rnejad/global-sufficiency/tree/main. our main contributions are: although tcav can be applied to all neural network classifiers, for simplicity we limit our experiments to binary roberta-based abusive language classifiers. we choose the roberta-based models for their superior performance in processing social media data compared to other base language models @xcite . the concept, c, is defined by n c concept examples. also, n r random examples are used to define non-concept examples. the roberta representations for all these examples are calculated using f emb , which maps an input text to its [cls] token representation. then, p number of cavs, υ p c , are generated, each through training a linear classifier that separates a sub-sample (with size n c ) of concept examples from a sub-sample of random examples (with size n r ) in the roberta embedding space. the conceptual sensitivity of a label to the cav, υ p c , at input x can be computed as the directional derivative s c,p (x): in this work, we use two metrics to specify the influence of the concept on the model's prediction. first, we calculate t cav dir , the fraction of inputs in a set of input examples x, for which the directional derivative s c,p (x) is positive, i.e.: t cav dir indicates the fraction of input examples for which the prediction scores of the model increase if the input representation is infinitesimally moved towards the concept representation. this metric has been widely used to identify if the label has learned the concept as an important signal for the label @xcite . besides the widely used metric of t cav dir (referred to as t cav score in previous work), we introduce a new metric, t cav mag , which considers the size of the directional derivatives, and measures the magnitude of the influence of the concept on the label for the positive directional derivatives: we demonstrate in our results that t cav mag can be an indicator of the over-reliance of the label on the concept. when calculated for all cavs, equations 2 and 3 generate two distributions of scores with size p for the concept c. using a t-test, these distributions are compared with the distributions of t cav dir and t cav mag calculated for random examples to check for statistical significance @xcite . ## false global sufficiency phenomenon p is considered globally sufficient for phenomenon q (p ⇒ q) if, whenever p occurs, q also occurs @xcite . in other words, global sufficiency refers to the extent to which a concept can explain the model's output across all instances in a held-out dataset, as opposed to the more studied topic of local sufficiency, which concerns the stability of an individual prediction for a given feature in perturbed contexts @xcite . in a real-world setting, it is very unlikely that any single concept is truly sufficient for the label at a global level. in a binary classifier, a concept c is falsely learned as sufficient for the positive label if all inputs containing c are classified as positive by the classifier, regardless of context. this undesired dependency of the label on the concept suggests that the model has failed to learn how the concept interacts with context to influence the label. while this issue is closely related to spurious correlation, we use the term false global sufficiency because spurious correlation typically implies that the feature is irrelevant to the label, and a correlation is learned due to a confounding factor. in contrast, we consider the cases where the feature is relevant and important but not globally sufficient. to make this clearer, consider the case of abusive language detection and the concept of negative emotions; if the mere presence of negative emotions in a sentence always guarantees the prediction of the positive label (abuse), then the model has learned a false sufficiency relation between the concept and the label. it over-relies on this feature and ignores the context. to quantify falsely learned global sufficiency, we consider two scenarios: 1) where a balanced challenge set is available, which contains c in all of its examples (both classes), and 2) where no challenge set is available. for the first scenario we use the traditional approach of assessing accuracy of the classifier on a held-out test set. this approach provides a baseline in our evaluations. for the second scenario, we propose concept-based explanation metrics and compare them with the baselines obtained with the challenge sets. ## sufficiency of the concept of describing protected groups with negative emotion in this section, we evaluate the metrics introduced in section 3 in explaining the extent of the falsely learned sufficiency between a human-defined concept and the positive label of the classifiers. we specifically consider the concept of describing a protected group with negative emotion words and refer to it as desnegem for brevity. we chose this concept because it is tightly related to hate speech and is expected to be important for more general definitions of harmful language, such as toxic, abusive or offensive. still, it is not a sufficient concept for these labels and has to be interpreted in the broader context (as shown by examples in we consider three roberta-based binary classifiers, publically available and trained with large english datasets. the models are trained for general definitions of abusive language, toxicity or offensive language. we refer to these classifiers by their training datasets: jigsaw, civil comments (or civil for brevity) and tweeteval. these models are described in detail in appendix a quantifying sufficiency with a challenge set: to calculate the metric described in section 3.1, we first use the hatecheck @xcite test cases to build a challenge set for the concept of desnegem. for that, we use the f2 and f21 functionalities of hatecheck, i.e., the hateful and non-hateful examples that include this concept ( we obtain f alse_suf f of 0.41, 0.40, and 0.50 for civil, jigsaw and tweeteval, respectively. this metric shows a higher falsely learned sufficiency score for tweeteval than the two other classifiers, as expected. based on these observations, we expect tcav metrics to show lower scores for civil and jigsaw than the tweeteval classifier. global sufficiency with concept-based explanations: here, we use the results obtained with the challenge set as a baseline to evaluate the tcavbased metrics. concept examples are generated using the template '<@xmath0.', @xmath1, trans people, gay people, black people, disabled people, muslims and immigrants as identified by @xcite . @xmath2, we use the disgust and anger categories of the nrc emotion intensity lexicon (nrc-eil) @xcite . we use the nltk package 3 to filter out words other than adjectives, past tense verbs and past participles, and also remove the words with emotion intensity lower than 3 https://www.nltk.org/ 0.5. after these steps, we are left with 368 concept words. we calculate the t cav dir and t cav mag scores for the concept of desnegem and compare those to the metrics calculated for random concepts with t-test for statistical significance. for random concepts, the concept examples are random tweets collected with stop words. in our implementation of the tcav procedure, n @xmath3, n @xmath4, n @xmath5. for input examples, x, we use 2000 tweets collected with stop words. as presented in discussion: for all classifiers, the presence of the concept describing a protected group with negative emotion words is a strong signal for the label of abuse. all classifiers struggle in considering the broader contexts in sentences such as 'it is not acceptable to @xmath6.' among the three classifiers, tweeteval has learned a higher degree of sufficiency, leading to its worse performance on a challenge set containing this con- ## global sufficiency of fine-grained negative emotions concepts in the previous section, we considered the concept of describing protected groups with negative emotions, which is tightly related to hate speech, and thus prone to be mistakenly learned as sufficient for the label of abuse. in this section, we test our proposed method for a less obvious case by disentangling the concept of emotions and hate speech. we focus on the concept of describing a (non-protected) group of people with negative emotions, which differs from the previous section in 1) removing the protected groups and replacing them with unprotected groups and 2) breaking down the emotion concept to more fine-grained levels. for fine-grained emotion concepts, we first develop a compact challenge set, examples of which are presented in for the challenge set, we write five abusive and five non-abusive example templates for each emotion. then we generate 40 abusive and 40 non-abusive examples by @xmath7, chinese people, doctors, teachers, school children, football players, my neighbours, and men to represent non-protected groups. 4 full list of examples of this challenge set is available in our github repository mentioned in section 1. equivalently, for the tcav procedure for concept templates, we use 'they @xmath8, instead of '<@xmath9, which we used in section 4. ## conclusion concept-based explanations can assess the influence of a concept on a model's predictions. we used two metrics based on the tcav method: the tcav direction score identifies whether the classifier has learned an association between a concept and a label, and the tcav magnitude score measures the extent of the influence of the concept on the label. we showed that the best-performing abusive language classifiers learned that negative emotion is associated with abuse (positive direction) but did not over-rely on this concept (low magnitude); that is, they did not overestimate the global sufficiency of that concept. our method can potentially be used for other nlp classification tasks. this approach is suitable for tasks where certain concepts are closely related to the label, but not enough to make a definitive determination. for example, in sentiment analysis, the price of products may have a strong connection to negative sentiment, but is insufficient to determine it. further research should explore how concept-based explanations can help identify cases where certain concepts are relied upon too heavily in abusive language detection or other nlp classification tasks.