Thesis Distillation: Investigating The Impact of Bias in NLP Models on Hate Speech Detection (2023)

Abstract: This paper is a summary of the work done in my PhD thesis. Where I investigate the impact of bias in NLP models on the task of hate speech detection from three perspectives: explainability, offensive stereotyping bias, and fairness. Then, I discuss the main takeaways from my thesis and how they can benefit the broader NLP community. Finally, I discuss important future research directions. The findings of my thesis suggest that the bias in NLP models impacts the task of hate speech detection from all three perspectives. And that unless we start incorporating social sciences in studying bias in NLP models, we will not effectively overcome the current limitations of measuring and mitigating bias in NLP models.

## introduction hate speech on social media has severe negative impacts, not only on its victims @xcite but also on the moderators of social media platforms @xcite . this is why it is crucial to develop tools for automated hate speech detection. these tools should provide a safer environment for individuals, especially for members of marginalized groups, to express themselves online. however, recent research shows that current hate speech detection models falsely flag content written by members of marginalized communities, as hateful @xcite @xcite . similarly, recent research indicates that there are social biases in natural language processing (nlp) models @xcite @xcite @xcite . yet, the impact of these biases on the task of hate speech detection has been understudied. in my thesis, i identify and study three research problems: 1) the impact of bias in nlp models on the performance and explainability of hate speech detection models; 2) the impact of the imbalanced representation of hateful content on the bias in nlp models; and 3) the impact of bias in nlp models on the fairness of hate speech detection models. investigating and understanding the impact of bias in nlp on hate speech detection models will help the nlp community to develop more reliable, effective, and fair hate speech detection models. my research findings can be extended to the general task of text classification. similarly, understanding the origins of bias in nlp models and the limitations of the current research on bias and fairness in nlp models, will help the nlp community develop more effective methods to expose and mitigate the bias in nlp models. in my thesis and this paper, i, first, critically review the literature on hate speech detection ( §2) and bias and fairness in nlp models ( §3). then, i address the identified research problems in hate speech detection, by investigating the impact of bias in nlp models on hate speech detection models from three perspectives: 1) the explainability perspective ( §4), where i address the first research problem and investigate the impact of bias in nlp models on their performance of hate speech detection and whether the bias in nlp models explains their performance on hate speech detection; 2) the offensive stereotyping bias perspective ( §5), where i address the second research problem and investigate the impact of imbalanced representations and co-occurrences of hateful content with marginalized identity groups on the bias of nlp models; and 3) the fairness perspective ( §6), where i address the third research problem and investigate the impact of bias in nlp models on the fairness of the task of hate speech detection. for each research problem, i summarize the work done to highlight its main findings, contributions, and limitations. thereafter, i discuss the general takeaways from my thesis and how it can benefit the nlp community at large ( §7). finally, i present directions for future research ( §8). the findings of my thesis suggest that the bias in nlp models has an impact on hate speech detection models from all three perspectives. this means that we need to mitigate the bias in nlp models so that we can ensure the reliability of hate speech detection models. additionally, i argue that the limitations and criticisms of the currently used methods to measure and mitigate bias in nlp models are direct results of failing to incorporate relevant literature from social sciences. i build on my findings on hate speech detection and provide a list of actionable recommendations to improve the fairness of the task of text classification as a short time solution. for a long-term solution to mitigate the bias in nlp models, i propose a list of recommendations to address bias in nlp models by addressing the underlying causes of bias from a social science perspective. the main contributions of this survey are: 1) there are different definitions and forms of hate speech. one of the main limitations of current studies on hate speech detection, is the lack of distinction between hate speech and other concepts like cyberbullying. 2) there are many resources of hate speech related datasets in the literature, that allow the development of new hate speech detection models. however, these datasets have many limitations, including limited languages, biased annotations, class imbalances, and user distribution imbalances. 3) one of the main limitations of the current research on hate speech detection, is the lack of understanding how it is impacted by the bias in nlp models. this limitation is what i aim to address in my thesis. ## survey: hate speech limitations: one of the main limitations of this survey, is that it focuses on hate speech detection only as a supervised text classification task. however, recent studies propose a framework to automate and enforce moderation policies, instead of training machine learning models to detect hate speech @xcite . similarly, this review focuses on hate speech datasets that are collected only from social media platforms. however, recently, generative models have become more popular and started to be used in generating hate speech related datasets @xcite . ## survey: bias and fairness in nlp in @xcite , i review the literature on the definitions of bias and fairness in nlp models. additionally, i review the literature on the origins of bias in nlp models from two perspectives: 1) nlp pipeline as discussed in @xcite ; hovy and prabhumoye (2021), and 2) social sciences and critical race theory as discussed in @xcite ; @xcite . there are many definitions of the term bias. the normative definition of bias, in cognitive science, is: "behaving according to some cognitive priors and presumed realities that might not be true at all" @xcite . and the statistical definition of bias is "a systematic distortion in the sampled data that compromises its representatives" @xcite . the statistical definition of bias is the one used in this thesis. in this work, i argue that the sources of bias in the nlp pipeline originate in the social sciences and that they are direct results of the sources of bias from the social science (jim code) perspective as shown in figure 1 . the main contribution of this literature review is reviewing the sources of bias in nlp models from the social science perspective as well as the nlp perspective. this survey points out the limitations of the currently used methods to measure and mitigate bias in nlp models. it also suggests that these limitations are direct results of the lack of inclusion of social science literature in the development of methods that quantify and mitigate bias in nlp. finally, i share a list of actionable suggestions and recommendations with the nlp community on how to mitigate the limitations discussed in studying bias in nlp ( §7). limitations: one main limitation of this survey is that it reviews the literature on the sources of bias in the nlp pipeline, only in supervised models. unsupervised nlp models might have different sources of bias. the second limitation is regarding the reviewed literature on the sources of bias in social sciences, where i rely mainly on three books algorithms of oppression: how search engines reinforce racism by safiya nobel (nobel, 2018), race after technology: abolitionist tools for the new jim code by ruha @xcite , and more than a glitch: confronting race, gender, and ability bias in tech by meredith broussard @xcite . a more comprehensive literature review to review studies that investigate the direct impact of social causes on bias in nlp would be important future work. however, to the best of my knowledge, this area is currently understudied. in the next sections, i address the understudied impact of bias in nlp models on hate speech detection models. i investigate that impact from the following perspectives. ## the explainability perspective for this perspective, i investigate the performance of different hate speech detection models and whether the bias in nlp models explains their performance on the task of hate speech detection. to achieve that, i investigate two sources of bias: based on these findings, i investigate whether the social bias resulting from pre-training contextual word embeddings explains their performance on hate speech detection in the same way syntactical bias does. i inspect the social bias in three lms (bert (base and large) @xcite , albert (base and xx-large) @xcite , and roberta (base and large) @xcite ) using three different bias metrics, crows-pairs @xcite , stereoset @xcite , and seat @xcite , to measure gender, racial and religion biases. first, i investigate whether large models are more socially biased than base models. the wilcoxon statistical significance test @xcite indicates that there is no statistical significant difference between the bias in base and large models in bert and roberta, unlike the findings of @xcite . however, there is a significant difference between the base and xx-large albert. these results suggest that large models are not necessarily more biased than base models, but if the model size gets even bigger, like albert-xx-large, then the models might get significantly more biased. since there is no significant difference between the base and large models, i only use base lms in the rest of the thesis. then, i follow the work of @xcite and use correlation as a measure of the impact of bias on the performance of the task of hate speech detection. the pearson's correlation coefficients between the bias scores of the different models and the f1-scores of the different models on the used five hate-speechrelated datasets are inconsistently positive as shown in figure 2 . however, due to the limitations of the metric used to measure social bias, as explained in @xcite , the impact of the social bias in contextual word embeddings on their performance on the task of hate speech detection remains inconclusive. based on these findings, i inspect the impact of social bias, gender, and racial, in the swe on their performance on the task of hate speech detection. to measure the social bias in the swe, i use the following metrics from the literature: weat @xcite , rnsb @xcite , rnd @xcite , and ect @xcite . then, i use pearson's correlation to investigate whether the social bias in the word embeddings explains their performance on the task of hate speech detection. similar to lms, the results indicate an inconsistent positive correlation between the bias scores and the f1-sores of the bi-lstm model using the different word embeddings as shown in figure 3 . this lack of positive correlation could be due to limitations in the used metrics to measure social bias in swe @xcite . these results suggest that the impact of the social bias in the swe on the performance of the task of hate speech detection is inconclusive. contributions: the main findings and contributions of the explainability perspective can be summarized as: 1) the results provide evidence that the syntactical bias in contextual word embeddings, resulting from pre-training, explains their performance on the task of hate speech detection. limitations: one of the main limitations of this work is using social bias metrics from the literature, which have their limitations as argued in @xcite ; @xcite . additionally, the work done here, is limited to hate speech datasets that are in english. similarly, the social bias inspected in the different word embeddings is based on western societies, where the marginalized groups might be different in different societies. it is also important to mention that the findings of this work are limited to the used datasets and models and might not generalize to other models or datasets. in @xcite ; elsafoury (2023), i investigate how the hateful content on social media and other platforms that are used to collect data and pre-train nlp models, is being encoded by those nlp models to form systematic offensive stereotyping (sos) bias against marginalized groups of people. especially with imbalanced representation and co-occurrence of the hateful content with the marginalized identity groups. i introduce the systematic offensive stereotyping (sos) bias and formally define it as "a systematic association in the word embeddings between profanity and marginalized groups of people." @xcite . i propose a method to measure it and validate it in static @xcite and contextual word embeddings @xcite . finally, i study how it impacts the performance of these word embeddings on hate speech detection models. i propose the normalized cosine similarity to profanity (ncsp) metric, which is a metric to measure the sos bias in static word embeddings using the cosine similarity between a list of swear words and non-offensive identity (noi) words that describe three marginalized groups (women, lgbtq, and non-white) described in word embeddings mean sos women lgbtq non-white w2v 0.293 0.475 0.456 glove-wk 0.435 0.669 0.234 glove-twitter 0.679 0.454 0.464 ud 0.509 0.582 0.282 chan 0.880 0.616 0.326 glove-cc 0.567 0.480 0.446 glove-cc-large 0.318 0.472 0.548 ft-cc 0.284 0.503 0.494 ft-cc-sws 0.473 0.445 0.531 ft-wk 0.528 0.555 0.393 ft-wk-sws 0.684 0.656 0.555 sswe 0.619 0.438 0.688 debias-w2v 0.205 0.446 0.471 p-desip 0.266 0.615 0.354 u-desip 0.266 0.616 0.343 sos_lm scores biased towards profanity biased towards nice sentences bert-base no-bias race gender sexual-orientation religion disability social-class biased towards profanity biased towards nice sentences albert-base no-bias race gender sexual-orientation religion disability social-class biased towards profanity biased towards nice sentences roberta-base no-bias word embeddings. the results show that, for the inspected static word embeddings, the correlation results, according to pearson correlation, show a negative correlation between the measured sos bias scores measured using the ncsp metric and the social bias scores (gender and race) measured using the weat, rnd, rnsb, and ect metrics. as for the contextual word embeddings, the pearson correlation coefficient results show a positive correlation between the sos bias scores measured using the sos lm metric and the social bias scores (gender, race, and religion) measured using the crows-pairs metric, which could be the case because the sos lm metric is built on the crows-pairs metric. finally, i investigate whether the inspected sos bias explained the performance of the inspected word embeddings on the task of hate speech detection. i train mlp and bi-lstm models with an untrainable layer of the different static word embeddings on four hate-speech-related datasets. as for contextual word embeddings, i fine-tune bert-base-uncased, albert-base, and roberta-base on six hate speech related datasets. then, i use pearson's correlation between the sos bias scores in the different word embeddings and their f1 scores on the models on the task of hate speech detection. the correlation results, similar to the results in §4, show an inconsistent positive correlation. this could be because the limitations of other social bias metrics in the literature are extended to the proposed metrics. in this case, the impact of the sos bias in static and contextual word embeddings on their performance on the task of hate speech detection remains inconclusive. contributions: the main findings and contributions of the offensive stereotyping perspective can be summarized as follows: 1) i define the sos bias, propose two metrics to measure it in static and contextual word embeddings, and demonstrate that sos bias correlates positively with the hate that marginalized people experience online. limitations: the findings of this work are limited to the examined word embeddings, models, and datasets, and might not generalize to others. similarly, the sos bias scores measured using the ncsp metric in the inspected static word embeddings, are limited to the used word lists. another limitation is regarding my definition of the sos bias, as i define bias from a statistical perspective, which lacks the social science perspective as discussed in blodgett et al. (2021); @xcite . moreover, i only study bias in western societies where women, lgbtq and non-white ethnicities are among the marginalized groups. however, marginalized groups could include different groups of people in other societies. i also only use datasets and word lists in english, which limits our study to the english-speaking world. similar to other works on quantifying bias, our proposed metric measures the existence of bias and not its absence @xcite , and thus low bias scores do not necessarily mean the absence of bias or discrimination in the word embeddings. another limitation of this work is the use of template sentence-pairs to measure the sos bias in contextual word embeddings, which do not provide a real context that might have impacted the measured sos bias. since the proposed method used to measure the sos bias in contextual word embeddings (sos lm ) builds on social bias metrics like crows-pairs and stereoset, it is highly likely that sos lm have the same limitations as crows-pairs and stereoset that are pointed out in @xcite . ## the fairness perspective in @xcite , i investigate how different sources of bias in nlp models and their removal impact the fairness of the task of hate speech detection. improving the fairness of the text classification task is very critical to ensure that the decisions made by the models are not based on sensitive attributes like race or gender. i first measure three sources of bias according to @xcite : representation bias, selection bias, and overamplification bias. then, i fine-tune three language models: bert, albert, and roberta on the jigsaw dataset @xcite , and measure the fairness of these models using two sets of fairness metrics: threshold-based and thresholdagnostic. the threshold-based metrics are the tpr_gap and the fpr_gap metrics used in @xcite ; @xcite . as for the threshold-agnostic metric, i use the auc_gap metric, which is an adaptation of the metrics proposed in @xcite . i investigate the impact of the different sources of bias on the models' fairness by measuring the pearson correlation coefficient between the bias scores and the fairness score. then, i investigate the impact of removing the three sources of bias, using different debiasing methods, on the fairness of hate speech detection models. i remove the representation bias using the sentdebias method proposed in @xcite to remove gender, racial, religious and sos bias on the inspected language models. to remove the selection bias, i aim to balance the ratio of positive examples between the identity groups in the jigsaw dataset. to achieve that, i generate synthetic positive examples using existing positive examples in the jigsaw training dataset, but with word substitutions using the nlpaug tool that uses contextual word embeddings to generate word substitutions @xcite . to remove the overamplification bias, i aim to ensure that the different identity groups, in the jigsaw dataset, appear in similar semantic contexts in the training dataset, as proposed in @xcite . to achieve that, i use different methods: 1) create data perturbations, 2) i use the sentdebias method to remove the bias representations from the finetuned models. thereafter, i compare the fairness of the inspected language models on the task of hate speech detection before and after removing each of the inspected source of bias. i aim to find the most impactful source of bias on the fairness of the task of hate speech detection and to find out the most effective debiasing method. the results suggest that overamplification and selection bias the results in contributions: the main findings and contributions of the fairness perspective can be summarized as follows: 1) the results demonstrate that the dataset used to measure the models' fairness on the downstream task of hate speech detection plays an important role in the measured fairness scores. limitations: it is important to point out that the work done in this section is limited to the examined models and datasets. this work studies bias and fairness from a western perspective regarding language (english) and culture. there are also issues regarding the datasets that those metrics used to measure the bias @xcite . the used fairness metric, extrinsic bias metrics, also received criticism @xcite . this means that even though i used more than one metric and different methods to ensure that our findings are reliable, the results could be different when applied to a different dataset. it is also important to mention that there is a possibility that the findings regarding the most effective debiasing method, which is finetuning the models on a perturbed dataset, is the case because i use a perturbed fairness dataset as well. i recognize that the provided recommendations to have a fairer text classification task rely on creating perturbations for the training and the fairness dataset. it might be challenging for some datasets, especially if the mention of the different identities is not explicit, like using the word "asian" to refer to an asian person but using asian names instead. additionally, for the sentiment analysis task, the used keyword to filter the imdb dataset and get only gendered sentences might provide additional limitations that might have influenced the results. moreover, in this section, i aim to achieve equity in the fairness of the task of text classification between the different identity groups. however, equity does not necessarily mean equality, as explained in @xcite . in this section, i combine all the findings of my thesis and point out how this work can benefit the nlp community and the ongoing research on hate speech detection, bias, and fairness in nlp. the survey of the literature on hate speech detection in §2 shows a lack of research on the impact of bias in nlp models and hate speech detection models. especially the impact on the performance of hate speech detection, and how the hateful content led nlp models to form an offensive stereotyping bias, in addition to limitations with the current research that investigates the impact of bias in nlp models on the fairness of hate speech detection models. the aim of my thesis is to fill these research gaps. the research goal of my thesis is to investigate the bias in nlp models and its impact on the performance and fairness of the task of hate speech detection, and more generally, the task of text classification. the findings of my thesis show that the bias in nlp models is preventing us from having reliable and effective hate speech detection and text classification models. this is evident by the findings of my thesis. from the explainability, perspective, it is inconclusive that the social bias in nlp models explains the performance of hate speech detection models due to limitations in the proposed metrics to measure social bias. however, the results in §4 also indicate that the bias resulting from pre-training language models, e.g., syntactic bias and biased pre-training datasets, impacts and explains their performance on hate speech detection modes. this good performance suggests that the hate speech detection model associates hateful content with marginalized groups. this might result in falsely flagging content written by marginalized groups on social media platforms. from the offensive stereotyping bias perspective, the findings in §5 demonstrate that word embeddings, static and contextual, are systematic offensive stereotyping (sos) biased. the results show no strong evidence that the sos bias explains the performance of the word embeddings on the task of hate speech detection, due to limitations in the proposed metrics to measure the sos bias. however, the existence of sos bias might have an impact on the hate speech detection models in ways that we have not explored or understood yet, especially against the marginalized groups. from the fairness perspective, the findings of §6 show that the inspected types of bias, representation, selection, overamplification, have an impact on the fairness of the models on the task of hate speech detection, especially the downstream sources of bias which are selection and overamplification bias. this means that the bias in the current hate speech datasets and the bias in the most commonly used language models have a negative impact on the fairness of hate speech detection models. hence, researchers should pay attention to these biases and aim to mitigate them before implementing hate speech detection models. these findings assert the notion that bias in nlp models negatively impacts hate speech detection models and that, as a community, we need to mitigate those biases so that we can ensure the reliability of hate speech detection models. however, in §3, i discuss the limitations and criticisms of the currently used methods to measure and mitigate bias in nlp models that fail to incorporate findings from the social sciences. as a short-term solution to improve the fairness of hate speech detection and text classification tasks, i provide a list of guidelines in @xcite . these guidelines can be summarized as follows: ## future work in this section, i discuss important future research directions to mitigate the limitations of this work and the literature on nlp.