T oxi G en: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022)

Abstract: Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language. To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model. Controlling machine generation in this way allows ToxiGen to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of ToxiGen and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human-written data substantially. We also demonstrate that ToxiGen can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset.

## introduction toxic language detectors often over-rely on minority identity mentions foot_0 when flagging a statement as toxic, without considering the deeper semantic meaning of the statement @xcite . this can lead to severe underdetection of subtle hate (e.g., "they have been bred to be good at sports and entertainment, but not much else"; figure 1 ) and over-detection of benign statements (e.g., "child abuse is wrong, racism is wrong, sexism is wrong"; figure 1 ). importantly, such biases in toxicity detection risk further marginalizing or censoring minority groups @xcite @xcite @xcite . we introduce toxigen, a large-scale machinegenerated dataset of 274,186 toxic and benign statements. to create this dataset, we leverage the massive pretrained language model gpt-3 @xcite , which is known to produce closeto-human-like text @xcite but also easily generates socially biased and toxic content @xcite . while such human-like bias and toxicity poses real threats, we use this undesirable behavior in models like gpt-3 to improve existing toxic language classifiers, providing a path forward for mitigating systemic bias. created using demonstrationbased prompting and pretrained toxicity classifiers, toxigen covers over 135k toxic and 135k benign statements about 13 minority identity groups (e.g., african americans, women, lgbtq+ folks, etc.). using this machine generated approach has two advantages over scraping posts from the web as done by previous work (e.g., @xcite @xcite . first, it allows us to limit spurious identity-toxicity correlations @xcite by generating equal numbers of toxic/benign statements for each demographic group, including those that are often overlooked in toxic language corpora (e.g., native americans). second, machine generation and careful prompting enables us to generate implicit toxicity (i.e., without swearwords or slurs), which is by definition hard to detect or find and thus often missing in toxic language corpora @xcite . indeed, 98.2% of toxigen statements are implicit, i.e., devoid of explicit profanity, generator classifiers alice ## implicit hate against minority groups detecting implicit toxicity about minority groups (e.g., stereotyping, microaggressions), remains an elusive goal for nlp systems @xcite . one key challenge is that, in contrast to explicit toxicity, implicit toxicity is not marked by the use of profanity or swearwords, is sometimes positive in sentiment, and is generally harder to detect or collect at scale @xcite . nonetheless, implicitly toxic language about minority or marginalized groups is often psychologically damaging to members of those groups @xcite ; datasets properties source size % implicit % hate class breitfeller et al. (2019) reddit 2,934 99.4 100.0 tweetblm (kumar and pranesh, 2021) twitter 9,165 99.0 33.7 de gibert et al. (2018) stormfront 9,916 92.2 11.3 waseem (2016) twitter 16,914 82.4 31.7 implicithatecorpus (elsherief et al., 2021) twitter 22,584 96.8 39.6 davidson et al. (2017) twitter 24,802 30.2 5.0 @xcite hate forums 27,665 71.8 9.1 dynahate @xcite human-machine adv. 41,134 83.3 53.9 socialbiasframes @xcite social media 44,671 71.5 44.8 founta et al. (2018) twitter 80,000 26.1 7.5 toxigen (ours) gpt-3 274,186 98.2 50.1 a second challenge for detecting subtle toxicity about minority groups is that minority mentions are more often the targets of social biases and toxicity @xcite . as such, minority mentions often co-occur with toxicity labels in datasets scraped from online platforms @xcite . for example, over 93% of mentions of jewish folk in @xcite are toxic @xcite . in turn, models trained on such data can exploit these spurious minority-toxicity correlations instead of considering the deeper semantics of text @xcite . importantly, the spurious correlations are also learned by large language models, which are known to produce stereotypical, biased, or toxic content when prompted with minority mentions @xcite . given that the main mitigation approach to prevent large language models (llm) from generating toxic language is to train new classifiers to detect such language, these classifiers also learn the spurious correlations and start blocking most language referencing minority groups. this risks erasure @xcite . with toxigen, we aim for generating a large scale dataset that represent implicit toxicity while balancing between toxic and benign statements, to address the gaps of previous work. as shown in ## creating toxigen to create toxigen, we use demonstration-based prompting for llms, encouraging a text generator to produce both toxic and benign sentences that mention minority groups without using explicit language. we introduce a classifier-in-the-loop decoding method based on constrained beam search, alice, which, along with samples generated without alice, contributes to generating a challenging subset of toxigen. using these methods, we generate a massive set of statements (over 274,000) containing equal numbers of toxic and benign sentences for 13 identity groups-see ## human validation of toxigen to ensure the quality of toxigen, we conduct human validation experiments and create toxigen-humanval, a human-validated test set. specifically, we investigate the reliability of our promptbased and alice-based methods at generating human-like statements and controlling statements' toxicity and the minority groups mentioned ( §4.2). additionally, we measure the effectiveness of al-ice-generated statements (vs. top-k-generated) at fooling classifiers ( §4.3). ## improving toxicity classifiers to further showcase the usefulness of toxigen, we investigate how it can enhance classifiers' abilities to detect human-written and machinegenerated implicit toxic language. we fine-tune the widely-used hatebert @xcite and toxdectroberta @xcite models on the training portion of toxigen, using the prompt labels as proxies for a true toxicity label. then, we compare the performance of the out-ofthe-box models to those fine-tuned on toxigen on three publicly available human-written datasets (implicithatecorpus @xcite , the socialbiasframes test set @xcite , and dynahate @xcite ) as well as the evaluation portion of our machine-generated dataset (toxigen-humanval). to ablate the contribution of each decoding method, we also split toxigen into equal numbers of alice-generated and top-k-generated examples. our results-see ## conclusions in this work, we used a large language model to create and release toxigen, a large-scale, balanced, and implicit toxic language dataset. toxigen is far larger than previous datasets, containing over 274k sentences, and is more diverse, including mentions of 13 minority groups at scale. the generated samples are balanced in terms of number of benign and toxic samples for each group. we proposed alice, an adversarial decoding scheme to evaluate robustness of toxicity classifiers and generate sentences to attack them, and showed the effectiveness of alice on a number of publicly-available toxicity detection systems. in our experiments, we showed that fine-tuning pre-trained hate classifiers on toxigen can improve their performance on three popular human-generated toxicity datasets. we also conducted a human study on a subset of toxigen, verifying that our generation methods successfully create challenging statements that annotators struggle to distinguish from human-written text: 90.5% of machine-generated examples were thought to be human-written.