Towards Procedural Fairness: Uncovering Biases in How a Toxic Language Classifier Uses Sentiment Information (2022)

Abstract: Previous works on the fairness of toxic language classifiers compare the output of models with different identity terms as input features but do not consider the impact of other important concepts present in the context. Here, besides identity terms, we take into account high-level latent features learned by the classifier and investigate the interaction between these features and identity terms. For a multi-class toxic language classifier, we leverage a concept-based explanation framework to calculate the sensitivity of the model to the concept of sentiment, which has been used before as a salient feature for toxic language detection. Our results show that although for some classes, the classifier has learned the sentiment information as expected, this information is outweighed by the influence of identity terms as input features. This work is a step towards evaluating procedural fairness, where unfair processes lead to unfair outcomes. The produced knowledge can guide debiasing techniques to ensure that important concepts besides identity terms are well-represented in training datasets.

## introduction previous nlp works have studied the fairness of toxicity detection classifiers by comparing the distributions of prediction scores across different demographic groups as input features @xcite . however, other toxicityrelated concepts are often present in the text and affect the differences in score distribution between identity groups. here, we introduce a framework that uses concept-based global explanations to uncover unintended biases for different identity groups, while controlling for a certain toxicityrelated concept. to demonstrate the effectiveness of concept-based explanations in uncovering biases, we specifically focus on sentiment, although the general methodology can be applied to any other relevant human-defined concept. negative sentiment is a salient toxicity feature, which has been used in designing feature-based and neural toxicity detection systems @xcite @xcite , and highly correlates with toxic language when targeted at demographic groups. assessing the differences in score distributions for various demographics is an example of outcome fairness. in fact, most fairness criteria used in machine learning measure outcome fairness, such as accuracy parity (equal accuracy for protected and unprotected groups), equality of opportunity (equal true positive rates), or equalized odds (equal true positive and false positive rates) @xcite . while valuable, outcome fairness metrics are costly to compute as they require large labelled datasets and do not provide any information about the model's decision making processes. more recently, work has begun to focus on the complementary notion of process fairness (also known as procedural fairness), or the idea that the decision-making process itself must be fair. @xcite conducted one of the first studies on process fairness in machine learning, measuring the extent to which people believed it was permissible to use various features as input to a criminal recidivism prediction algorithm. for example, they found that people generally felt that criminal history was fair to use as an input feature, but that it was unfair to use family criminality as input. another aspect of process fairness is that the importance given to an attribute in the decision-making process shouldn't be very different for different demographic groups. an example of this is the recent sffa vs. harvard court case where it was argued that academic and extracurricular achievements of asian-american applicants are given less weight in the admissions process compared to their white-american counterparts @xcite . we take a similar view of process fairness and consider a classifier as unfair if it either ignores or over-utilizes a feature for some demographic groups compared to others. in the current nlp landscape, one major barrier to assessing process fairness is that predictive models rarely use human-understandable concepts as input features, and so it is increasingly difficult to understand what high-level features foot_0 are actually being learned and used by the classifier. in this work, we use an interpretability framework of concept-based explanations @xcite , which enables us to explain a machine learning model's decision-making via conceptual units understandable to humans. concept-based explanations have been studied mostly in the context of computer vision, where it is fairly straightforward to define concepts of interest with a set of representative examples. however for textual data, it is much less clear how to define a concept in an effective and intuitive manner, and global explainability methods that operate on high-level abstractions remain under-explored @xcite . @xcite define a concept to be a meaningful, human-defined abstraction, which is expected to be important for the task at hand and which can be specified by a coherent set of examples. following this definition, we identify sentiment as a concept for toxicity classification. 2 to the best of our knowledge, this is one of the first works to apply concept-based explanations to the domain of nlp, and the first one to explore its effectiveness in identifying high-level fairness issues in models that work with textual data. in this work, we show how to use concept-based explanations to determine whether a trained toxicity classifier uses the information of sentiment as an important feature in its predictions. for that we use a multi-class model, described in section 2, and compare the importance of the concept of sentiment in predicting different subtypes of toxicity. although intuitively, negative sentiment should be an important signal for toxicity detection, its presence is neither necessary nor sufficient for an utterance to be tagged as toxic. for example, "muslims are grieving" carries a negative sentiment but is not abusive, whereas "you are so smart for a woman" is perceived as an insult despite including a positive sentiment word. also, sentiment might not be a distinguishing feature for some variations or subtypes of toxic language, such as threats or cyberbullying. for all the classes of our multiclass model, we ask, "has the classifier learned the concept of sentiment as a coherent and important high-level feature associated with this label?", and answer this question with concept-based explanations (section 5). we then assess how the presence of identity terms impacts the use of sentiment information by the classifier. for that, we control the context for sentiment and ask if the learned sentiment information is used similarly and fairly across identity groups (section 6). our code and data is available at https://github.com/isarnejad/p rocedural-fairness-sentiment. ## multi-class toxicity model for our experiments, we use an open-source, roberta-based model 3 (hanu, 2020) trained on the english dataset released as part of a kaggle competition on identifying and reducing bias in toxicity classification of online comments. 4 . the dataset includes public comments from the civil comments platform manually annotated for toxicity as well as six toxicity subtypes: severe toxicity, obscene, identity attack, insult, threat, and sexual explicit. the values for each label represent the fraction of the annotators that assigned the label to the comment. there are over 1.8m examples in the training set and around 195k examples in the test set. we exclude the class severe toxicity from our experiments, since there are only eight training examples with values higher than 0.5 for this class. further, a subset of the data is annotated for various identity groups mentioned in the text. the most frequently mentioned identity groups include male, female, homosexual (gay or lesbian), christian, jewish, muslim, black, white, people with psychiatric or mental illness. the classification model optimizes the competition's official evaluation metric that combines the overall auc with bias aucs for the identity groups @xcite . for this, the model's loss function combines the weighted loss functions for two tasks, toxicity prediction and identity prediction. this simple and straight-forward model has been shown to effectively reduce bias on non-toxic sentences that mention identity terms, and results in a competitive score of 93.74 on the test set. we chose this model for two reasons. first, the model is publicly available and is trained on one of the largest available toxicity dataset, annotated for multiple types of toxicity. an alternative choice for our experiments would be using multiple toxicity classifiers. however, the definitions of subtypes of toxicity are usually ambiguous and similar labels might be used for different subtypes of toxicity across datasets. in the case of our multi-class model, the disparities in using sentiment information can be reliably attributed to differences in subtype definitions. second, the model is debiased to some extent with regards to outcome fairness metrics. uncovering biases in such a model highlights the issue that optimizing for outcome fairness does not guarantee the procedural fairness in decision making. ## sentiment lexicon to formalize sentiment concepts, we employ the nrc valence, arousal, and dominance (nrc-vad) lexicon (mohammad, 2018), which provides manually annotated real-valued scores of valence, arousal, and dominance for 20,000 english words. we use the valence scores and convert them into the range from -1 (the most negative) to 1 (the most positive). we automatically select single words from the lexicon that are predominantly used as adjectives in the british national corpus (bnc) foot_4 and sort them in decreasing order by their frequency in the bnc. the n most frequent adjectives that can be used to describe humans or groups of humans are manually selected as the sentiment words to define the sentiment concepts. the sentiment range [-1, 1] is divided into five intervals: very negative [-1, -0.75], negative (-0.75, -0.25), neutral [-0.25, 0.25], positive (0.25, 0.75), and very positive [0.75, 1]. for each interval, @xmath0. 6 these sets of adjectives are then used to populate the sentence templates to define the sentiment concepts as described in section 5. ## concept-based explanations concept-based explanation is an emerging area in black-box model explainability, aiming to explain neural network models at the abstraction level defined by a human user @xcite . most explainability methods provide importance weights for low-level input features such as pixels in images or tokens for text @xcite @xcite . however, a user might want to evaluate the model's functionality at the level of a concept that is expected to be important for the model's prediction, which can be achieved with concept-based explanations @xcite . @xcite states that a concept needs to satisfy the properties of meaningfulness, coherency and importance for the task at hand. some examples of concepts in computer vision tasks are the concept of stripes for the class of zebra @xcite , the concept of white coat for the class of doctor @xcite , and the concept of nuclei texture in the detection of tumor tissue in breast lymph node samples @xcite . in the case of text, @xcite used conceptbased explanations to measure the sensitivity of a abusive language classifier to the emerging concept of covid-related anti-asian hate speech, and @xcite explained a text classifier with respect to the concepts identified through topic modeling. here, our goal is to explain the prediction of a toxicity classifier at the level of sentiment information learned by the trained model. since sentiment is not one of the direct input features of the model, feature importance metrics such as lime @xcite and shap (lundberg and lee, 2017) cannot be used to provide its importance. instead, we consider each level of sentiment as a concept and calculate the importance of these concepts for the model's predictions with testing concept activation vectors (tcav) algorithm. in the following section, we explain the tcav algorithm in detail. ## classifier sensitivity to sentiment in this section, we analyze the sensitivity of the classifier described in section 2 and identify the classes for which the classifier is sensitive to sentiment as a coherent and important feature. for the concepts of sentiment level, we create concept examples and use the tcav technique to test the importance of the concepts for each class. to create concept examples for each level of sentiment described in section 3, we use the template "they @xmath1.". this simple template ensures minimal extra semantic information other than sentiment level and avoids the problem of encoding unwanted biases in the concept itself, as described by tong and kagal (2020). it is important to note that such minimal templates cannot be labeled for toxicity without more context. consider a sentence such as "they are terrible.". this sentence expresses a negative sentiment but lacks any other significant information. only with more context can we say whether this sentence is toxic or not. the statement "these people are immigrants. they are terrible." is toxic, while the comment "i don't like these computers. they are terrible." is non-toxic, and "i don't like these singers. they are terrible." can be toxic depending on the specific definition of toxicity in a use case. therefore, the fairness analysis methods that rely on labels cannot be used to study the impact of these templates on the model's predictions. in addition to five levels of sentiment, we use two control concepts with predictable sensitivities: 1) a non-coherent concept, defined by a set of random tweets collected with stop words, for which we expect low average tcav scores for all labels; we observe that the tcav scores for the control concepts are as expected-zero sensitivity for a non-coherent, random concept and high sensitivity to the concept of explicit offence for the labels toxicity, obscene, insult and sexual explicit. for the sentiment concepts, we observe that the classifier is sensitive to very negative and negative sentiment for the labels toxicity and insult. 9 we also observe that the classifier is not sensitive to the neutral, positive and very positive sentiment concepts for any of the labels, which rules out the sensitivity of the classifier to the specific sentence structure of the templates. literature suggests that a high tcav score indicates: 1) the concept is learned by the classifier as a coherent feature, and 2) that feature is important for the classifiers' predictions @xcite . we evaluate the tcav scores shown in high average tcav scores indicate a significant increase of prediction probabilities when the concept is added to random contexts. we append the concept examples to random tweets and measure the prediction probabilities before and after the addition of the concept examples. the average increase of probabilities for all concepts and labels is shown in ## sensitivity to sentiment towards an identity group in the previous section, we demonstrated how the tcav framework can be used to assess whether a human-defined concept is learned by a classifier as an important feature. with that we showed that for some labels our model is sensitive to the presence of very negative and negative sentiments in broader contexts. here, we turn to the concept of "associating a sentiment with an identity group" foot_9 and ask if similar levels of sensitivity to sentiment are observed in the presence of certain demographic terms as input features. for creating the concept examples, we use the template "<@xmath2, @xmath3: women, gay people, trans people, muslims, immigrants, black people, and disabled people. we also add the terms these people and these things, to assess the sensitivity of the model to the concepts of "associating a sentiment with people in general" and "associating a sentiment with objects" as two baselines. as expected, we observe that the classifier is not sensitive to any level of sentiment when associated with objects. we discuss some of the most salient results below. we first assess the influence of identity terms by analysing the classifiers' sensitivities to the neutral sentiment towards the identity groups. figure 1 visualizes the column of table a.1 related to the neutral sentiment. from the findings in figure 2 visualizes the results of table a .1 from a different perspective. in this figure, we assess the sensitivity of the classifier to different levels of sentiment across the identity groups. for the relevant classes, we expect to see that the classifier is sensitive to negative sentiment but not sensitive to positive sentiment; i.e., the average tcav score should be high for negative sentiment and low for positive sentiment. consistent with results in another anomalous result with respect to the negative sentiment is that the classifier is sensitive to negative and very negative sentiment for the class sexual explicit, but only in the context of the identity group gay people. additionally, for the label of threat the classifier shows some sensitivity to the very negative and negative sentiment, but only when used with the identity term muslims. we then turn to assessing the sensitivities to positive sentiment. in general, the expected pattern of sensitivities is only observed for the labels toxicity and insult and the identity term women. for other cases, as the level of sentiment changes from very negative and negative to positive and very positive, the sensitivity of the classifier does not decrease as expected and remains high, specifically for identity groups gay people, black people and muslims. this observation indicates that a sentence such as "black people are wonderful" in a conversation will increase the probability of that text being labeled as toxic, identity attack and insult. also, for sexual explicit the classifier is sensitive to the mention of gay people for all levels of sentiment. we conclude that the classifier is oversensitive to the presence of these identity terms regardless of the level of sentiment, and even highly positive sentiment cannot cancel out the impact of the identity terms. one exception is the term disabled people, for which the model is under-sensitive, i.e, not sensitive to even negative and very negative sentiments associated with this group. ## discussion our results demonstrate that while a multi-class toxicity classifier generally shows high sensitivities to negative sentiment for certain classes, and zero sensitivities to neutral or positive sentiment, the picture changes when the sentiment is applied to certain marginalized identity groups. then, counterintuitively, even positive sentiment can increase the probability of a toxicity label. this suggests an over-reliance on the identity group term. previous work in computer vision has underscored the difficulty in finding unbiased examples with which to define concepts, e.g., when searching for images of men or women to examine gender bias, the examples invariably also contain information about age, race, and so on. here, in the context of nlp, we propose a generalizable solution to that problem, by generating examples rather than collecting them, and carefully controlling the variable of interest (here, sentiment, although the method could extend to other features). for this we use existing lexicons, without the need to label the examples for the various toxicity classes, as would be required for an analysis of outcome fairness. this knowledge of how the model uses the sentiment information can guide debiasing techniques. for example, a data augmentation approach can ensure important features are present in the training dataset. in the case of our model, a data balancing procedure should collect and label positive and very positive sentiments associated with gay people, black people and muslims, as well as very negative sentiments associated with disabled peo-ple and add them to the training dataset. it is important to note that evaluating models for the sensitivity to human-defined concepts is a tool to reveal flaws of a trained model, where prior knowledge about expected sensitivities is available. similar to previous test suits such as hatecheck @xcite , our method should not be considered as a standalone evaluation of models. moreover, observing the expected sensitivities does not guarantee the fairness of the model. only where unexpected sensitivity patterns are observed, the biases can be detected and mitigated accordingly. our method has limitations. we carry our analysis for one grammatical construction that expresses the concept of associating sentiment to identity groups. future work is needed to assess the generalizability of our results to other expressions of sentiment. moreover, tcav requires access to at least some model layers and cannot be applied when the model itself is unavailable. ## related work identifying and mitigating unintended biases in nlp systems to ensure fair treatment of various demographic groups has been focus of intensive research in the past decade @xcite . various metrics to quantify biases in system outputs have been proposed, including group fairness metrics and individual fairness metrics @xcite . however, to apply such metrics, the datasets need to be annotated with demographic attributes, which is costly and sometimes infeasible to do (e.g., the demographics of the authors of social media posts are often unknown). alternatively, the bias metrics are applied on synthetic data automatically generated using simple templates @xcite . in both cases, the test data are limited, and the evaluation is restricted to a set of pre-defined contexts. explainability techniques (xai) can potentially help in discovering and quantifying biases. much work on xai has been motivated by the need to assist in bias detection and mitigation @xcite . however, only a handful of nlp studies have actually employed explainability methods for bias detection and to a limited extent @xcite @xcite . @xcite surveyed the works at the intersection of fairness and xai in nlp and discussed conceptual and practical challenges in applying current explainability approaches for debiasing nlp models. multiple outlined issues stem from the fact that most current xai methods employed in nlp provide explanations on a local level through post-hoc processing, and it is still an open question how to generalize these local explanations to reveal systematic model biases. the tcav framework used in this paper produces global explanations and can therefore uncover unfair processes in the model's decision making. probing classifiers are well-known interpretability tools used to examine the encoded information in the representation layers of nlp models @xcite . probes are trained independently from the original model to predict an externally defined property (e.g., linguistic properties such as part of speech) from the model's representations. despite being widely used, several studies revealed that probes are not well-controlled, and caution should be taken when drawing behavioural conclusions about the original model from the performance of probing classifiers @xcite . also, probes can only assess whether the information about the property of interest is encoded in the representations (e.g., @xcite ) but do not provide evidence about how the model uses this information. to that end, extensions of probing classifiers were proposed, which assess the effect of removing the property's information with counterfactual interventions to provide causal explanations and mitigate biases in nlp classifiers @xcite . however, several concerns are raised about the effectiveness and the unintended consequences of removing attributes @xcite . while causality-driven probing methods assess the necessity of the property for the classifier's decision, tcav determines whether the model uses the encoded information as an important signal for a particular class. also, tcav allows us to quantify the relative importance of different properties encoded in the representation, which is not feasible with probing classifiers. the tcav framework has been developed and mostly applied in image classification. in the original paper, @xcite showed how gender and racial biases can be discovered with tcav in image classifiers. @xcite extended the method to regression problems, and applied it to detect gender and first language biases in automatic spoken language assessment. tong and kagal (2020) studied the effectiveness of tcav in discovering gender biases in image classification and discussed the difficulties in obtaining quality examples to represent a concept while not introducing new sources of bias (e.g., introducing a racial bias when selecting gendered examples). adhikari (2021) used tcav to measure gender bias when classifying faces as young or old, and discussed the difficulty of defining 'disentangled' concepts that only encode the concept of interest. to the best of our knowledge, our work is the first in applying tcav to discover biases in text classifiers. ## conclusion building on previous studies that measured group fairness in toxic language detection, this work is a step toward a more systematic and fine-grained analysis of procedural fairness in neural model's predictions. we use a global explainability metric to uncover the disparities in how the classifier learns to associate identity terms with domainrelevant concepts, e.g. sentiment. future work will focus on extending the analysis to other concepts known to be important to toxic language detection (profanity, threats of violence, dehumanizing or othering language, and so on) as well as additional classifiers, domains, and types of bias.