Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models (2023)

Abstract: Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic content. To address this challenge, we propose two novel pretraining data augmentation strategies that significantly reduce model toxicity without compromising its utility. Our two strategies are: (1) MEDA: adds raw toxicity score as meta-data to the pretraining samples, and (2) INST: adds instructions to those samples indicating their toxicity. Our results indicate that our best performing strategy (INST) substantially reduces the toxicity probability up to 61% while preserving the accuracy on five benchmark NLP tasks as well as improving AUC scores on four bias detection tasks by 1.3%. We also demonstrate the generalizability of our techniques by scaling the number of training samples and the number of model parameters.

## introduction pretrained large language models (lms) have become indispensable for solving various nlp tasks @xcite @xcite , yet it has been challenging to safely deploy them for real world applications @xcite . they have been known to generate harmful language encompassing hate speech, abusive language, social biases, and threats @xcite @xcite . these harmful generations are broadly referred to as "toxicity". 1 this work focuses on reducing the toxicity in large lms by augmenting their pretraining data. prior work has primarily focused on reducing toxicity either by finetuning lms on non-toxic data @xcite @xcite or using decoding time algorithms to re-weight the probabilities of toxic words @xcite @xcite . these methods typically incur further costs of finetuning additional lms @xcite , generating large amount of non-toxic data @xcite , or procuring human feedback @xcite . these techniques are known to reduce toxicity but also compromise perplexity and downstream task performance @xcite . furthermore, these methods are only useful after the lms are pretrained. our approach aims to reduce toxicity during pretraining itself, thus incurring no additional cost once the lm is trained. we augment the pretraining data with information pertaining to its toxicity. we believe that instead of filtering toxic data @xcite , the toxicity information can guide the lm to detect toxic content and hence generate non-toxic text. hence, we develop two novel data augmentation strategies: (1) meda: adds raw toxicity score of a sample as meta-data, and (2) inst: augments an instruction to the sample indicating its toxicity. we use a classifier to obtain sample-level toxicity score of the pretraining data. these scores are used by meda and inst to educate the lm about toxicity. fig. 1 shows an example of how meda and inst are applied. we add the raw toxicity score of the sample along with a tag "toxicity: 0.1" for the meda strategy. similarly, we add an instruction such as "this is a non-toxic post. post:" to the tokens of a non-toxic sample for inst strategy. no data augmentation is applied for base. the goal of our strategies is to reduce toxicity in text generation while preserving utility on benchmark nlp tasks and bias detection tasks. prior work only evaluates the success of toxicity reduction techniques on realtoxici-typrompts @xcite . few techniques are evaluated for their utility in performing some benchmark nlp tasks @xcite . but toxicity reduction techniques like data filtering can reduce the bias and toxicity detection capabilities of the lms @xcite . some techniques like finetuning @xcite can also reduce the capability of the lm to effectively perform downstream end-to-end text generation tasks. hence, we expand the evaluation to include -(1) bias detection tasks: we evaluate the capability of our strategies to detect social biases @xcite , and (2) text generation task: we measure the performance of our strategies on the e2e task @xcite . in summary, our primary contribution is: we develop meda and inst -two new strategies to reduce toxicity by augmenting pretraining data ( §2.2). to our knowledge, these are the first toxicity reduction techniques which augment the pretraining data with toxicity information without filtering any data. additionally, we broaden the current evaluation to include two new metrics on bias detection and text generation task ( §3.1). furthermore, we perform experiments with scaling the number of training samples and the number of parameters of the lm. our results demonstrate that our best performing strategy (inst) considerably reduces the toxicity probability of the generations by as much as ∼ 61% while preserving the utility of the lm on five benchmark nlp tasks as well as improving on the four bias detection tasks by 1.3% ( §4). moreover, we demonstrate that our strategies applied at samplelevel perform better than document-level @xcite by 11% in toxicity probability reduction ( §6). ## methodology our approach guides the lm about the toxicity of the data it sees during training and directs it to generate non-toxic content. we educate the lm by augmenting the pretraining dataset d with toxicity information. we first use a classifier to obtain toxicity scores for samples (s) in d. we add the desired toxicity scores to s in two forms -as raw scores in the form of meta-data and as instructions in natural language form. ## experimental setup modeling details we train all our models from scratch using the decoder-only architecture in megatron-lm @xcite . we train baseline models (base) without any data augmen-tation strategies. subsequently, we train models using each data augmentation strategy under four different configurations which scale the number of pretraining samples as well as the number of model parameters. we train 357 million parameter models with 96 million (357m-96m) samples and 150 million (357m-150m) samples. similarly, we train 1.3 billion parameter models with 96 million (1.3b-96m) and 150 million (1.3b-150m) samples. 3 for all models, we use @xmath0.5, @xmath1.1 and @xmath2.9. this implies 90% of toxic samples in the dataset are appended with c tox . note that only 4.14% of the entire train set has @xmath3.5 (fig. 2 ) which means that 3.73% (90% of 4.14) of the entire train set receive c tox . for meda @xmath4.5 and for inst @xmath5.9. 34.59% of the entire dataset has @xmath6.1 (from fig. 2 ). this means that for meda 17.3% of samples and for inst 31.13% of samples are augmented with c nont . from these values, we derive that for meda ∼ 79% of samples and for inst 65.14% of samples don't undergo any modification. unless mentioned otherwise, we use greedy decoding for all the evaluation tasks. ## results and discussions through this section we present the aggregated results and analyze them. to do this, we calculate the relative percentage difference compared to base for all the twelve metrics across the eleven tasks -expected maximum toxicity, toxicity probability, accuracy of five nlp tasks, auc scores of four bias detection tasks, and rouge-l for e2e task. we then compute an average across all the metrics (we also include the experiments with control variable c nont ). in fig. 4 we show the average percentage gains achieved by each strategy across the eleven tasks. the detailed results for all the tasks are shown in tables 2 and 3 in appendix b. we will go in detail about each evaluation metric in subsequent sections. fig. 4 demonstrates that all the strategies are better than base. fig. 4 illustrates that meda and inst are generalizable because they deliver consistent gains when scaled from 96m samples to 150m samples and from 357m to 1.3b model parameters. if we average the gains across the four models, then we observe that inst strategy attains the most gains (14%) and hence is the best since the performance of the strategies is consistent across the four models, we only show the average behavior of the approaches across the four models for each of the metrics. toxicity evaluation fig. 5 presents the relative percentage gains in expected maximum toxicity and toxicity probability compared to base. it also shows the results for meda and inst by using their respective control variable c nont vs not. we observe that meda and inst are successful in reducing the toxicity of generations as evaluated by realtoxicityprompts setup @xcite . specifically, we see huge gains in toxicity reduction when we use control variable c nont associated with meda and inst (compare striped vs non-striped bars for each color). this is because we are directing the lm to generate non-toxic content by prefacing the prompt with c nont . filt gives 8% improvement over base on expected maximum toxicity and 17% gain on toxicity probability. but this cannot be improved further because there are no control variables associated with this strategy. inst on the other hand establishes 29.3% improvement in expected maximum toxicity and a 61.3% improvement in toxicity probability i.e inst reduces the probability of generating toxic content by ∼ 61% and the probability is as low as 0.14. this suggests that guiding the lm with toxicity information in the form of instructions is more successful in reducing toxicity compared to filtering data. benchmark nlp tasks fig. 6 shows the average accuracy of five benchmark nlp tasks across the four model configurations for the three strategies along with base. we observe that meda and inst are as competent as base on the nlp tasks. in fact, we observe a marginal gain @xmath7, meda and inst. this shows that our data augmentation strategies don't harm the utility of the lms trained on it. bias detection tasks additionally, fig. 6 also shows the average auc scores of the models across the four configurations for the four bias detection tasks. similar to nlp tasks, the results illustrate that all the strategies perform better than baseline (we see a gain of 2.5% for filt, 1.4% for meda, and 1.3% for inst). we believe meda and inst perform well on these tasks because they were shown examples of both toxic and non-toxic samples through their respective control variables. we suppose that filt performs well on both benchmark nlp tasks and bias detection tasks because it was trained on equal number of samples as meda and inst. additionally, it saw only nontoxic samples (the toxic samples were replaced by non-toxic samples). hence, the perplexity for toxic sentences would be higher in filt. we discuss this in detail in §6. text generation task since we see huge gains in fig. 5 by adding c nont , we want to evaluate if adding c nont affects the performance of a downstream text generation task. fig. 7 shows the average of rouge-l scores across the four model configurations for meda and inst strategy using c nont (striped bars) and without using it (nonstriped bars). fig. 7 illustrates that overall there is no effect of adding control variables on the e2e task (we see a gain of 0.5% and 2.0% for meda and inst respectively when using c nont ). ## related work finetuning-based methods pretrained lms can be further finetuned using different training algorithms like domain-adaptive training methods @xcite @xcite and reinforcement learning @xcite on non-toxic data. these methods can only be employed after lms are pretrained. these methods typically incur further costs of finetuning additional lms @xcite , generating large amount of non-toxic data @xcite , or procuring human feedback @xcite . our work on the other hand is targeted towards reducing toxicity by augmenting the pretraining corpus and hence will not incur additional cost after the lm is trained. decoding time algorithms they reduce toxicity of the generations at decoding time by altering the probabilities of certain tokens. gehman et al. (2020) show a study on using pplm @xcite , word-filtering, and vocabulary shifting @xcite . @xcite use the internal knowledge of the lm to reduces the probability of generating toxic text. the gedi approach @xcite guides the generation at each step by computing classification probabilities for all possible next tokens. @xcite propose dexperts which controls the generation with an"expert" lm trained on non-toxic data and "anti-expert" lm trained on toxic data. these techniques are efficient at reducing toxicity but fail to consider the underlying semantic meaning of the generated text at the sequence level. they may also reduce the utility of the lm at performing downstream tasks @xcite . ## comparison with prior work prior work and this study uses different model configurations in terms of model parameters, pretraining data, number of samples, and hyperparameters. we show comparison with closest model configuration with our work. we only compare the relative changes because the baselines are different for our work and @xcite ; @xcite ; @xcite . also, perspectiveapi (perspec-tiveapi, 2022) update their models regularly and hence scores returned by it may change over time. @xcite develop sgeat which first generates large amount of non-toxic data using a pretrained lm @xcite and then uses domain adaptive finetuning. they have the same model parameters and pretraining data as our models. we follow the same toxicity evaluation setup and similar setup for benchmark nlp tasks. we compare 357m-150m inst model ( ## conclusion and future work we develop two new strategies to reduce toxicity using data augmentation -meda and inst. through extensive experiments, we demonstrate that meda and inst reduce toxicity probability substantially (54% and 61% respectively) while not compromising on the utility of the lm on five benchmark nlp tasks and four bias detection tasks. we also show that adding control variables does not compromise performance on e2e task. in this work, we show how toxicity can be reduced in lms by augmenting the pretraining data with toxicity information. we believe that this idea can be extended to other dimensions of social biases and hate speech. prior work shows that adding instructions during finetuning can help various nlp tasks and improve the lms capabilities to generalize for instructions on unseen tasks @xcite . we postulate that these observations can be applied to adding instructions to the pretraining data which can make inst generalizable to reduce different types of biases. the key idea of adding relevant information to the pretraining data via instructions can be applied more broadly and opens new directions for future work. future work can focus on controlling the generation by adding general instructions to the pretraining data. current work has applied meda and inst on binary view of toxicity i.e something is toxic or non-toxic. hence, another interesting direction is to explore the degrees of toxicity and incorporate it with meda and inst strategies. future work can also evaluate the generalizability and applicability of inst strategy on more text generation tasks.