Suum Cuique: Studying Bias in Taboo Detection with a Community Perspective (2022)

Abstract: Prior research has discussed and illustrated the need to consider linguistic norms at the community level when studying taboo (hateful/offensive/toxic etc.) language. However, a methodology for doing so, that is firmly founded on community language norms is still largely absent. This can lead both to biases in taboo text classification and limitations in our understanding of the causes of bias. We propose a method to study bias in taboo classification and annotation where a community perspective is front and center. This is accomplished by using special classifiers tuned for each community’s language. In essence, these classifiers represent community level language norms. We use these to study bias and find, for example, biases are largest against African Americans (7/10 datasets and all 3 classifiers examined). In contrast to previous papers we also study other communities and find, for example, strong biases against South Asians. In a small scale user study we illustrate our key idea which is that common utterances, i.e., those with high alignment scores with a community (community classifier confidence scores) are unlikely to be regarded taboo. Annotators who are community members contradict taboo classification decisions and annotations in a majority of instances. This paper is a significant step toward reducing false positive taboo decisions that over time harm minority communities.

## introduction members of a community rely on a shared language for communication, one which evolves naturally, in situ, and is shaped by the norms and mores of the community @xcite . norms promote entitativity, the perception of group identity @xcite . norms may be explicit and codified into law or so subtle that while members may be unable to specify them they can still recognize violations @xcite . utterances violating a these norms may be considered taboo from that community's perspective. 1 communities often self-regulate by censuring norm violating taboo language @xcite . this censuring may be subtle such as ignoring taboo utterances leaving the individual somewhat isolated and ineffective. self-regulation could also be explicit and even severe such as expulsion from the community. taboos are community-specific: perceptions of taboos in language use are influenced by community @xcite . utterances that are benign in one community might be taboo in another 2 . e.g., 'autistic' is considered derogatory on most of reddit, its use usually leads to censure. however, on the subreddit /r/wallstreetbets, it is not and is instead used as a self-descriptor by members. the importance of considering the author's community when studying taboo is emphasized in a recent critical survey of bias in nlp papers @xcite ; it urges us to understand how social hierarchy relates to language use. our research is a step in this direction. taboo utterance detection and biases: several papers especially from offenseval @xcite have led to the development of state of the art taboo classifiers for moderating online speech. alongside, an active research stream studies the presence of bias in taboo detection. biases, specifically having higher false positive rates for a minority community compared to the majority, have been detected particularly against african americans and also to some extent women @xcite @xcite @xcite . bias is clearly harmful as penalties over false positives could over time discourage/constrain minority participation on social media platforms, and in turn reinforce the majority's norms and values. limitations in bias assessment research: a limitation is that community language norms are rarely considered. while recent research on taboo detection discuss and illustrate the importance of community perspective @xcite @xcite we do not yet have bias detection methods that are firmly founded on community language norms. this methodological gap underlies not only the design of state-of-theart taboo detection tools but perhaps more crucially even in the methodologies for the study of bias in taboo detection. not only are the taboo annotated datasets largely devoid of community, culture or social contexts (i.e., characterization) @xcite , the study of bias itself needs strengthening with methods wherein a community perspective stays front and center, consistent with the urging in the survey @xcite . ## community centered approach for studying bias we build community-specific classification models (clcs) which capture the community's language. we use these to compute text alignment scores for assessing bias. a text alignment score is the classifier's confidence in deciding if the text has been generated by a member of the community or not. higher confidence scores imply greater alignment with the language norm's of the community. our intuition is as follows. given moderation and self-regulation in communities we expect taboo utterances to be infrequent in a community's discourse. such utterances will thus have low alignment scores when classified by a model trained on the community's language. however, since new topics of interest will also result in low-frequency utterances we view low alignment scores with a community model as necessary but insufficient for an utterance to be considered taboo from that community's perspective. crucially, utterances with high alignment scores adhere to the norms and therefore should not be considered taboo. reference classifier models allow us to estimate the extent of alignment with community language norms. two points to note. we do not attempt to separate style from topic in the utterances since it is often in their interplay that taboo is decided. for example, "happy birthday to a bad bitch" and "fuck off, you are a little bitch." have somewhat related styles but different topics. our 'contextual' community-language classification models estimate scores for full utterances. second, while we study race/ethnicity based communities, our approach can be applied to communities defined broadly, using other criteria such as professional communities and internet subcultures. specific details for measuring bias are given next. measuring bias in taboo classifiers: we assess the extent of taboo classifier bias against a community by computing pearson correlation between the taboo classifier confidence scores and community classifier confidence scores. we do this with instances that the former classifier declares as taboo. ideally, we expect a negative correlation -higher taboo classifier confidence mapping to lower community alignment scores and vice versa. the extent to which correlations deviate from this expectation reflects the extent to which the classifier does not consider the norms of the community. measuring bias in taboo datasets: given a dataset, we compute the proportion of taboo labelled texts that are highly aligned with each community classifier model. we expect these proportions to be tending towards zero since high alignment means common utterance and thus within the norms. a second point to note is that if these proportions are uneven across communities then the bias is more against those with larger proportions. this is consistent with the theory that all communities engage in similar norms of communication @xcite . specifically, we measure the mean and standard deviations (sd) of proportion of aligned comments. in sum, if the proportions are not close to zero this indicates bias. additionally, if the proportions are not even then bias is targeting some communities more than others. ## related work the study of bias in taboo classifiers is an active area of research @xcite @xcite . since our focus is on bias detection we do not cover areas such as bias mitigation @xcite and allied problems such as bias in embedding spaces @xcite and the source of bias @xcite . instead we focus on bias detection -particularly on their methodologies. note as most papers combine methods our binning is not intended as mutually exclusive classes. bias detection focused on individual words: @xcite working with wikipedia talk pages show that toxicity classifiers disproportionately misclassify text with identity terms such as 'gay' and 'muslim' when they appear in benign contexts. a reason for this bias is because of their dominant occurrence in hateful, toxic or otherwise taboo contexts, thereby skewing training datasets. they show this can be countered by augmenting data with benign examples using these identity words. park et al. (2018), and kennedy et al. (2018) also study similar biases in the data. @xcite extend this analysis to words in general, finding strings like '@abc' and 'dirty' stereotypical of hate. the general approach in these papers is to compare classifiers built from skewed and synthetically augmented/corrected non-skewed datasets. in contrast to focusing on specific words, we analyze texts contextually with clcs. bias detection using an external race/gender labeled dataset: davidson et al. (2019) examine five datasets including the davidson and founta datasets we study. their strategy is to build a classifier (regularlized logistic regressor) on each dataset and test them on the blodgett et al. (2016) black-white-race-aligned dataset. they observed that 'black-aligned' tweets were 1.8 to 2.6 times more likely to be classified as taboo. using similar methodology, sap et al. (2019) also found bias against aa in neural network classifiers trained on the founta and davidson datasets when tested against the same blodgett et al. (2016) dataset. using a slightly different approach kim et al. (2020) trained a classifier on the blodgett et al. @xcite dataset to identify aa-leaning tweets in the founta dataset. they found that aa tweets were 3.7 times more likely to be labeled as taboo. additionally, they also annotated the founta dataset for gender. they found that aa male-aligned tweets were 77% more likely to be labeled as taboo. in contrast to these works, our approach does not rely on access to race labeled datasets such as the blodgett et al. (2016) dataset. other approaches: as in kim et al. (2020), sap et al. (2019) used classifiers trained on blodgett et al. ( 2016 ) to identify 'black-aligned' tweets in the founta and davidson datasets. the difference is that they then computed correlation of probabilities of 'black-aligned' tweets and the taboo language annotations and found these to be strong positive. we also compute pearson correlations between classifier scores and community alignment scores. but we differ in that we are not relying on the blodgett et al. (2016) dataset. instead, we use alignment scores estimated using clcs. while the above works highlight limitations in using a single standard english model they do not propose alternative methods founded on the language norms of specific communities. we address this gap with a community-specific method using clcs which allows us to study bias for any community. moreover, unlike the works reviewed, and more recent papers on debiasing strategies ( zhou et al. (2021); xu et al. (2021) ), where the emphasis is on bias against aa, we study bias against five different minority communities. ## limitations and conclusions we presented a new methodology for studying bias in taboo text identification. its strength is that it is centred on community language norms -a strategy consistent with @xcite to consider social hierarchy when studying bias and natural languages. using it we assessed the extent to which biases against five minority communities are present -both in classifier taboo decisions and in dataset taboo annotations. we found many instances of bias with the community most targeted being african americans. but we also found significant biases against others such as south asians. notably, hispanics seems least affected though there was to some degree classifier bias. a small scale 'illustrative' user study provides initial support for our key idea which is that common utterances, i.e., those with high alignment with the community's language classifier, are unlikely to be viewed as taboo from that community's perspective. annotators who are community members contradicted classifier and annotator decisions in a majority of instances. our work is limited to analyzing communities defined by race and ethnicity, but is generalizable; we will explore bias against other communities in future research. we also plan to conduct a largescale user study to better understand community perspective for taboo texts. we plan to investigate the bias mitigation strategies shown in section 4.5.