R eal T oxicity P rompts: Evaluating Neural Toxic Degeneration in Language Models (2020)

Abstract: Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning “bad” words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

## introduction although they are the backbone of many modern nlp systems @xcite @xcite , language models (lms) pretrained on large web text corpora suffer from degenerate and biased behavior @xcite . as illustrated in figure 1 , they can easily degenerate into toxicity, even without explicitly toxic prompts, which hinders their figure 1 : non-toxic examples from realtoxici-typrompts, a new testbed for evaluating neural generations and their toxicity. despite not containing any toxic language as measured by perspective api, these prompts cause several pretrained lms to systematically generate highly toxic text (shown in we first introduce a framework to systematically measure the risk of toxic degeneration by pretrained lms. we release realtoxici-typrompts ( §4), a set of 100k naturally occurring prompts (i.e., sentence prefixes; figure 1 ) extracted from a large corpus of english web text and paired with toxicity scores from a widely used and commercially deployed toxicity detector (perspective api). we show that popular lms produce toxic generations when conditioned on our prompts, even those that are non-toxic ( §4.2). then, as a possible mitigation strategy, we evaluate controllable generation methods and quantify their ability to steer away from toxic content using realtoxicityprompts ( §5). we find that certain controllable methods (e.g., toxicity control tokens, swearword filters) are less successful than more computationally or data-intensive methods (e.g., finetuning on non-toxic corpora). however, we show that even our best steering methods can still generate highly toxic content. finally, to further investigate the potential cause of these phenomena, we present the first largescale analysis of toxicity in gpt-2's training corpus, openai webtext, (openai-wt; @xcite , as well as an in-depth analysis of its open-source replica, openwebtext corpus (owtc; @xcite . we find non-negligible amounts of toxic, harmful, and abusive text in these corpora, which were used in pretraining of several language models (including roberta, ctrl, and gpt-2; @xcite . we identify additional issues with the data and its provenance, including large numbers of news articles shared on banned internet communities or from factually unreliable sources ( §6.2). our findings highlight the difficulty of avoiding toxicity in natural language generation (nlg) and illustrate a need to actively reconsider the content used in lm pretraining. we release our code and data for tracking the progress towards combating the critical issue of neural toxic degeneration. 1,2 2 operationalizing toxicity characterizing the toxicity of large corpora of naturally occurring or machine generated text is crucial to understanding toxic degeneration by language models. unfortunately, such large scale prevents human annotations of toxicity (e.g., we score at least 80 gb of text in §6). therefore, we rely on perspective api foot_2 , an automated tool for toxic language and hate speech detection. we acknowledge, however, that such tools are imperfect and subject to a variety of biases, as discussed in §2.2 and §7. ## out-of-the-box generation toxicity we focus our investigation of toxic degeneration in five popular autoregressive transformer-based @xcite language models: gpt-1, 5 https://github.com/conversationai/ perspectiveapi/blob/master/3-concepts/ score-normalization.md 6 to assess perspective api on human-generated text, the first three authors performed manual judgments of toxicity of a sample of 100 documents from owtc, and found an 88% pairwise agreement (pearson ⇢=0.83) with toxicity scores. to assess the api on machine-generated text, among 100 generations from gpt-2, our judgments had 80% pairwise agreement and pearson ⇢=0.65 with toxicity. for further model information, we refer the reader to the model card for tox-icity: https://github.com/conversationai/ perspectiveapi/blob/master/2-api/modelcards/english/toxicity.md gpt-2, gpt-3, ctrl, and ctrl-wiki. gpt-1 @xcite is a 117m-parameter model pretrained on a large corpus of english books @xcite . gpt-2 (specifically, gpt-2-small; @xcite , is a similarly sized model pretrained on openai-wt, which contains 40gb of english web text and is described in §6. 7 @xcite is pretrained on a mix of common crawl, an expanded version of openai-wt, books corpora, and wikipedia. 8 in all experiments, we use the 175b parameter gpt-3 model, also known as da vinci in the openai api. ctrl @xcite is a 1.63b parameter model that uses domain-specific control tokens for conditional language modelling. we analyze generations in two domains: web text (ctrl, links control token), and english wikipedia (ctrl-wiki, wiki control token). generating from models unless otherwise noted, we use nucleus sampling @xcite with @xmath0.9 to generate up to 20 tokens (see appendix §b.4 for additional details). all experiments are carried out with the hugging face transformers library @xcite . ## realtoxicityprompts to systematically evaluate and compare the generations from language models, we create real-toxicityprompts as a testbed for toxicity in conditional language generation that mirrors real world applications (e.g., autocomplete systems; @xcite . with this dataset, we quantify the effect of prompt toxicity on the toxicity of generation from our five language models. ## detoxifying generations we investigate the effectiveness of recent controllable generation methods at steering away from toxicity using realtoxicityprompts. specifically, we focus on gpt-2 as a base model for two detoxification techniques: data-based, where we pretrain the language model further, and decoding-based where we only change the generation strategy without changing model parameters. 11 as described in §4.2, we sample 25 generations per prompt for each model. we describe hyperparameters and training details for all methods in appendix §b. ## analyzing toxicity in web text to further investigate the phenomenon of neural toxic degeneration, and partially motivated by the surprising effectiveness of domain-adaptive pretraining on non-toxic data, we turn our focus to two corpora used to pretrain several language models. specifically, we quantify the toxicity in openai-wt (gpt-2's training data; @xcite and its open-source replica owtc @xcite , inspired by previous work in analyzing social biases in large text corpora @xcite . then, we investigate the provenance of the data in these corpora, quantifying how many documents come from factually unreliable news sites or were shared on quarantined or banned subreddits. owtc is a large corpus of english web text scraped from outbound urls in submissions on reddit communities (subreddits). in the creation of owtc, only links included in posts with a "karma" (i.e., popularity) score of 3 or more were considered. following the links, only english documents longer than 128 tokens are included in this corpus, amounting to 38 gb of text from about 8m documents. to allow for further analyses, we parse the urls given with owtc documents to extract the domain (often a news website, figure 5 in appendix §d; sharoff, 2020), which we crossreference with news factuality ratings by @xcite . we additionally cross-reference publicly 14 when releasing realtoxicityprompts, we will include a flag for prompts belong to this challenging subset. figure 3 : toxicity scores of documents in owtc (top) and openai-wt (bottom). y-axis is in log-scale, and color gradient follows magnitude in x-axis. we consider a document toxic if its toxicity is 0.5. we additionally display the estimated total % of toxic documents in each corpus above each subplot. available reddit dumps foot_6 to identify which subreddits the urls were submitted to. we include further details on owtc and metadata linking in appendix §d. openai-wt is the pretraining corpus for gpt-2 @xcite , also containing about 8m documents. following owtc, authors gathered urls from reddit, though from a different (but overlapping) timespan. additionally, authors filtered content using a blocklist of sexually-explicit and otherwise offensive subreddits. 16 this corpus does not come paired with url metadata. overlap we find about 29% overlap between the two corpora, using a large-scale similarity search with locality-sensitive hashing (rajaraman and ullman, 2011, see appendix d for details). we find that at least 2.3m documents in openai-wt also appear in owtc. ## discussion and recommendations overall, our investigations demonstrate that toxicity is a prevalent issue in both neural language generation and web text corpora. although they show some reduction in toxicity, steering methods do not fully protect neural models from toxic degeneration ( §5). additionally, the corpora that language models are pretrained on contain non-negligible amounts of toxic, abusive, and untrustworthy content ( §6). some implications of our findings are discussed below. effectiveness of "forgetting" toxicity our findings on data-based steering methods show that adaptive pretraining lowers a model's propensity to unpromptedly generate toxic language, but that its prompted generations can still be toxic. this raises the question: can language models ever fully "forget" toxic pretraining data through further adaptation @xcite ? the non-trivial amounts of toxicity generated by dapt suggest that perhaps language models may be "memorizing" the toxicity in pretraining data @xcite or that toxic examples may be more salient for the model and hence harder to unlearn @xcite . future work could explore whether some variants of toxicity are harder to forget than others, or whether the biases of models used to select training data for steering introduce unwanted side effects in language model behavior after adaptation. decoding with a purpose our analyses also highlight the promise of certain decoding methods, such as pplm @xcite , which is among the most effective methods we tested at avoiding toxicity with toxic prompts. in addition to automated toxicity classifiers, future work could explore the use of handpicked toxic documents as "negative examples" to avoid toxicity in generation. future work could also investigate infusing models with more sophisticated or nuanced representations of social biases @xcite . ## related work a wealth of work has shown that toxicity and social biases in training data are acquired by large pretrained sentence encoders (e.g., gender bias in bert; @xcite @xcite . however, fewer studies have investigated toxicity in autoregressive language models, whose generations also suffer from incoherence, blandness, and repetitiveness @xcite . similar in spirit to realtoxicityprompts, @xcite find universal adversarial triggers, nonsensical prompts that trigger toxic generations in gpt-2. in this work, we find and release naturally occurring prompts from web text that trigger toxicity, and compare toxic output in several language models. most closely related to this work, @xcite use a set of 60 templated prompts that mention majority or minority identities to study the social biases in generations by out-of-the-box pretrained language models. in our work, we study toxic degeneration by both out-of-the-box and controlled models using 100k naturally occurring prompts, including some that do not contain identity mentions (see figure 1 ). additionally, our work focuses on the broad phenomenon of toxicity in generations, whereas @xcite study the sentiment and regard expressed by a model's generation towards demographic identities. the creation of realtoxicityprompts was partly inspired by work in detecting conversational patterns that can cause derailment into antisocial behavior in online conversations @xcite @xcite . our work also draws from a strong line of research into controlling the outputs of language models @xcite @xcite . ## conclusion we introduce realtoxicityprompts, a testbed of 100k prompts for evaluating the toxic degeneration in pretrained language models. under this framework, we quantify the toxicity of multiple pretrained language models and the effectiveness of methods for detoxifying generations. we then analyze toxicity in two large web text corpora, including the gpt-2 pretraining corpus, to better understand the root cause of toxic generations. finally, we provide recommendations for gathering pretraining data. the data, code, and interactive visualizations for this paper can be found at https://toxicdegeneration.allenai.org/ .