Detecting Unintended Social Bias in Toxic Language Datasets (2022)

Abstract: With the rise of online hate speech, automatic detection of Hate Speech, Offensive texts as a natural language processing task is getting popular. However, very little research has been done to detect unintended social bias from these toxic language datasets. This paper introduces a new dataset ToxicBias curated from the existing dataset of Kaggle competition named “Jigsaw Unintended Bias in Toxicity Classification”. We aim to detect social biases, their categories, and targeted groups. The dataset contains instances annotated for five different bias categories, viz., gender, race/ethnicity, religion, political, and LGBTQ. We train transformer-based models using our curated datasets and report baseline performance for bias identification, target generation, and bias implications. Model biases and their mitigation are also discussed in detail. Our study motivates a systematic extraction of social bias data from toxic language datasets.

## introduction in the age of social media and communications, it is simpler than ever to openly express one's opinions on a wide range of issues. this openness results in a flood of useful information that can assist people in being more productive and making better decisions. according to statista 2 , the global number of active social media users has just surpassed four billion, accounting for more than half of the world's population. the user base is expected to grow steadily over the next five years. various studies (plaisime et al., 2020) say that children and teenagers, who are susceptible, make up a big share of social media users. unfortunately, this increasing number of social media users also leads to an increase in toxicity @xcite . sometimes this toxicity gives birth to violence and hate crimes. it does not just harm an individual; most of the time, the entire community suffers as due to its intensity. we have different perspectives based on race, gender, religion, sexual orientation, and many other factors. these perspectives sometimes lead to biases that influence how we see the world, even if we are unaware of them. biases like this can lead us to make decisions that are neither intelligent nor just. furthermore, when these biases are expressed as hate speech and offensive texts, it becomes painful for specific communities. while some of these biases are implied, most explicit biases can be found in the form of hate speech and offensive texts. the use of hate speech incites violence and sometimes leads to societal and political instability. blm (black lives matter) movement is the consequence of one such bias in america. so, to address these biases, we must first identify them. while the concepts of social bias and hate speech may appear to be the same, there are subtle differences. this paper expands on the above ideas and proposes a new dataset toxicbias for detecting social bias from toxic language datasets. the main contributions can be summarized as follows: in the following section we discuss various established works which are aligned with our work. section 3 provides information about our dataset, terminology, annotation procedure, and challenges. in section 3, we describe our tests and results, followed by a discussion of lexical overfitting reduction via data augmentation in section 5. section 6 discusses the conclusion and future works. in addition, there have been a lot of notable efforts towards detection of data bias in hate speech and offensive languages @xcite @xcite . @xcite has discussed the presence of unintended bias in hate speech detection models for identity terms like islam, lesbian, bisexual, etc. the biased association of different marginalized groups is still a major challenge in the models trained for toxic language detection @xcite . this is mainly due to the bias in annotated data which creates the wrong associations of many lexical features with specific labels @xcite . lack of social context of the post creator also affect the annotation process leading to bias against certain communities in the dataset @xcite . social bias datasets: more recently, many datasets @xcite have been created to measure and detect social biases like gender, race, profession, religion, age, etc. however, @xcite has reported that many of these datasets lack clear definitions and have ambiguities and inconsistencies in annotations. a similar study have been done in @xcite , where dataset has both categorical and free-text annotation and generation framework as core model. there have been few studies on data augmentation @xcite to decrease the incorrect association of lexical characteristics in these datasets. @xcite proposed a prompt based framework to generate large dataset of toxic and neutral statements to reduce the spurious correlation for hate speech detection. however, no study has been done for detecting social biases from toxic languages, which is a challenging task due to the conceptual overlap between hate speech and social bias. using a thorough guideline, we attempt to uncover harmful biases in toxic language datasets. the curated dataset is discussed in length in the next section, as are the definitions of each category label and the annotation procedure. ## toxicbias dataset we develop the manually annotated toxicbias dataset to enable the algorithm to correctly identify social biases from a publicly available toxicity dataset. below, we define social bias and the categories taken into account in our dataset. the comprehensive annotation process that we use for dataset acquisition is then covered. ## experimental setup in this section we will discuss about different models trained for detection of social biases and their categories. for all our experiment, we split the data into train, development, and test (80:8:12) set. since the dataset was imbalanced with respect to bias column, we split it in stratified manner. ## mitigation of model bias when we look at the incorrectly classified comments in ## conclusion and future work we have demonstrated that identity attacks or hate speech often incorporate social biases or stereotypes. however, not all hate speech can be labeled as social bias. some of them are merely personal insults. filtering out such biases from hate speech is not a trivial task. furthermore, we have frequently observed that detecting bias without context for the comment or demographic information of the comment holder makes the annotation much more challenging. however, detecting these social biases from toxic datasets, which are available in relatively large amounts, will be a useful starting point for social bias research in other forms of text. the issue of model bias is also observed during inference. the imbalanced existence of particular community terms (muslims, whites, etc.) might lead to a model labeling a comment as biased. to attenuate model biases, we used counter-narratives and showed that they help significantly to reduce model biases. from our study, we also observe that biases can have directions too. so basically, biases can occur against specific communities and in favour of a community. we intend to detect such biases in future work.