Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Pre-trained language models (LMs) are shown to easily generate toxic language.
Inthis work, we systematically explore domain-adaptive training to reduce the toxicityof language models.
We conduct this study on three dimensions: training corpus,model size, and parameter efficiency.
For the training corpus, we demonstrate thatusing self-generated datasets consistently outperforms the existing baselines acrossvarious model sizes on both automatic and human evaluations, even when it uses a13 smaller training corpus.
We then comprehensively study detoxifying LMs withparameter sizes ranging from 126M up to 530B (3 × larger than GPT-3), a scalethat has never been studied before.
We find that i) large LMs have similar toxicitylevels as smaller ones given the same pre-training corpus, and ii) large LMs requiremore endeavor to unlearn the toxic content seen at pre-training.
We also exploreparameter-efficient training methods for detoxification.
We demonstrate that addingand training adapter -only layers in LMs not only saves a lot of parameters butalso achieves a better trade-off between toxicity and perplexity than whole modeladaptation for the large-scale models.
Introduction
Large-scale pre-trained language models (LMs) [1–6] have demonstrated substantial performancegains on various NLP tasks, especially when scaling up the sizes of models.
However, recent studies [7, 8] show that LMs can generate toxic and biased language, which raises ethical concernsfor their safe deployment in real-world applications.
Previous methods on reducing the toxicity of LMs can be categorized as: decoding-time methods,pre-training-based methods and domain-adaptive training methods.
Decoding-time methods [9–14]manipulate the output distribution or input prompts at the inference stage without modifying theoriginal model parameters.
These methods can be flexible, but they either resort to some simpleword filtering strategies [10], or increase the computational cost at the inference stage.
For example,
PPLM [9] requires multiple iterations of backward propagation through the LM when generatingevery token, which makes it prohibitively expensive to be deployed to production especially forlarge-scale LMs.
In contrast, pre-training-based methods directly filter out the potentially toxiccontent within the pre-training corpus and retrain the model from scratch [e.g., 15].
However, it isdifficult to determine the filtering criterion beforehand, and pre-training a large LM multiple timesfrom scratch is quite expensive.
Domain-adaptive training methods [10, 16] further fine-tune the pre-trained LMs on carefully curateddatasets (e.g., Jigsaw, filtered OWTC [17]).
For instance, Gehman et al.
[10] construct a nontoxic data corpus from an existing dataset, OWTC, via the Perspective API 2 and perform the fine-tuning onthe nontoxic corpus.
Domain-adaptive training is more flexible than pre-training methods, as one canstill customize the model after the expensive pre-training process.
Compared to the decoding-timemethods, domain-adaptive training methods have the following advantages: i ) they can achieve fastand memory-efficient inference, thus can be deployed in broader systems; and ii ) they can largelyreduce the model toxicity while still maintaining good LM quality measured by perplexity anddownstream task performance as we will show in this work.
In this paper, we explore the limits of domain-adaptive training for detoxifying language models alongthe following three aspects: 1 ) Training Corpus : Unlike previous methods using curated pre-trainingcorpus for detoxification, we propose to leverage the generative power of LMs to generate nontoxiccorpus, which achieves better data efficiency for detoxification.
2 ) Model Size : We systematicallystudy and mitigate the toxicity issues in LMs with parameter sizes ranging from 126M to 530B,a scale that has never been studied before in this domain.
3 ) Parameter-efficient Training : Weinvestigate two parameter-efficient paradigm: adapter [18] and prefix-tuning [19], and compare themwith whole model adaptation in a systematic way.
We hope our work can shed light on the challengesof detoxifying large-scale LMs, as well as motivate the development of detoxification techniques thatare effective and parameter-efficient without significantly hurting the LM quality.
Summary of Contributions:
• We identify the trade-off between detoxification effectiveness (measured by Perspective API and
human evaluation) and language model quality (measured by validation perplexity and downstream task accuracy).
Existing approaches either suffer from limited detoxification effectiveness or significantly sacrifice the language model quality to detoxify LMs.
• We propose Self-Generation Enabled domain-Adaptive Training (SGEAT) that uses a self-
generated dataset for detoxification.
It mitigates the exposure bias [20, 21] from the discrepancy between teacher-forced domain-adaptive training and autoregressive generation at test time, andthus achieves better data efficiency.
In particular, we demonstrate that it consistently outperformsthe baseline approach with domain-adaptive training on pre-training data (DAPT) by a widemargin across various model sizes in terms of automatic and human evaluations, even whenwe use only a 13 smaller corpus for training.
By combining SGEAT with the state-of-the-artdecoding-time method, we can further reduce the toxicity of large-scale LM.• From the perspective of model size, we find that: i)
Large LMs have similar toxicity levels assmaller ones given the same pre-training corpus.
This implies the toxicity comes from the trainingdataset, instead of the model size.
ii)
Large LMs require more efforts (e g., larger training corpus)to reduce toxicity.•
We explore two parameter-efficient training methods for detoxification, and observe that: i)domain-adaptive training with adapter achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale LMs, and the improvement is more significant whenthe size of LMs increases; ii) prefix-tuning is less suitable for detoxification and demonstrateslimited detoxification effectiveness and perplexity control. We organize the rest of the paper as follows.
We discuss related work in § 2 and present our evaluationprotocols in § 3.
{+We then systematically explore the domain-adaptive training with respect to trainingcorpus in § 4, model sizes in § 5, and parameter efficiency in § 6. We present the human evaluationresult in § 7, and conclude the paper in § 8. Some text samples can be found in Appendix A.
+}  Related Work
Large-scale language models (LM) are pre-trained over a sizable collection of online data.
Theyare unavoidably exposed to certain toxic content from the Internet.
Recent studies [e.g., 22–24]show that pre-trained masked LMs display different levels toxicity and social biases.
Anotherline of work focuses on the toxicity of autoregressive LMs.
For instance, Wallace et al.
[8] firstdemonstrate that synthetic text prompts can cause racist continuations with GPT-2.
Gehman et al.[10] extend the analysis of LM toxicity to non-synthetic prompts, and create a benchmark dataset
R EAL T OXICITY P ROMPTS to provide a standard evaluation protocol via Perspective API to measure
LM’s toxicity, which is adopted by many previous work.
In this paper, we follow the standard settingto compare different detoxification approaches on different-sized LMs.
[Table]
Decoding-time methods They manipulate the decoding-time behavior of the LMs without changingthe model parameters [9–14].
Simple approaches such as word filtering and vocabulary shifting [10]directly lower the probability of toxic words (e [Table] g., swearwords, slurs, vulgar slang) being generated.
Though efficient, such approaches fail to consider the semantic meaning of the generated text at thesequence level.
Thus, it cannot completely prevent from generating toxic sentences which contain noundesirable words from the blocklist [15] ( e.g., “ poor people don’t deserve to live in nice houses ”).
Xu et al.
[13] perform sentence-level filtering by generating K continuations given the same promptand returning the most nontoxic sentence.
Similarly, Self-Debiasing [11] uses K manually craftedtemplates to manipulate the decoding probability distribution and dynamically set the probability oftoxic words to be low.
However, these methods lead to K times longer than the normal decoding.
PPLM [9] iteratively adds perturbation on the context vector at each step of decoding.
Though withbetter detoxification effectiveness, it suffers much more computational overhead due to multipleiterations of forwarding and backward propagation to generate the perturbations.
GeDi [12] guidesgeneration at each step with a second LM trained on nontoxic data by computing classificationprobabilities for all possible next tokens.
However, it requires an external LM trained on non-toxicdata, which is not easy to access in practice.
DE XPERT [14] controls the generation of large-scalepre-trained LM with an “expert” LM trained on non-toxic data and “anti-expert” LM trained ontoxic data in a product of experts [25].
It achieves the state-of-the-art detoxification results on
R EAL T OXICITY P ROMPTS , but sacrifices the validation perplexity and downstream task accuracy.
Domain-adaptive training methods They fine-tune the pre-trained LMs to the non-toxic domainby training on curated nontoxic data [10, 16, 26].
Gehman et al.
[10] use the DAPT framework[26] to further train LMs on the nontoxic subset (filtered via the Perspective API) of pre-trainingcorpus, OWTC, with GPT-2.
Besides DAPT, Gehman et al.
[10] propose to fine-tune on a corpuswith toxicity attribute token and prepend the nontoxic attribute token as prompt to yield nontoxicgeneration.
Solaiman and Dennison [16] propose a human-crafted Values-Targeted Datasets to changemodel behavior and reflect a set of targeted values.
In this work, we focus on exploring the limits ofdomain-adaptive training methods to reduce the toxicity of language models, while maintaining goodvalidation perplexity and downstream task accuracy.
Reinforcement learning (RL) methods There are two concurrent work [27, 28] that study thetoxcity behavior of LM with RL. InstructGPT [27] requires collecting human demonstrations andrankings of model outputs for two-stage fine-tunings.
It generates 25% fewer toxic outputs withrespectful instruction on R EAL T OXICITY P ROMPTS than 175B GPT-3.
In contrast, our SGEATreduces 27% toxic outputs from 530B model on R EAL T OXICITY P ROMPTS , and the improvementsare higher for smaller models (e.g., reduces 37% toxic outputs from 8B model).
To identify the toxic
LM behavior, Perez et al.
[28] uses RL to improve the generation of adversarial test cases.
Evaluation Protocols
In this section, we present our principle for evaluating different detoxification methods.
Specifically,we emphasize that detoxification method should focus on both reducing the model toxicity andmaintaining the model quality after detoxification.
We first discuss the protocol for LM toxicityevaluation, and then present the protocol to evaluate the LM quality before and after detoxification.
Pre-trained LMs.
We investigate the toxicity of a variety of standard GPT-3 like LMs with differentparameter sizes, ranging from 126M (similar to GPT-3 Small), 357M (similar to GPT-3 Medium),1.3B (similar to GPT-3 XL), 8.3B to the largest 530B [6]. All of the models are based on Trans-
[Figure]
former [29] with different hidden dimension, number of layers, and attention heads.
We present moredetails in Appendix §A.1.
All standard models are pre-trained on the same pre-training corpus, whichis an English text corpus constructed from 15 high-quality datasets.
3.1 Toxicity Evaluation
In this work, we follow prior work [15, 10] and perform both automatic evaluation and humanevaluation to measure an LM’s tendency to generate toxic language.
Automatic Evaluation relies on Perspective API, an online automated model for toxic language andhate speech detection.
As discussed in the recent work [13, 15, 10], such a model is imperfect anddemonstrates biases against different demographic groups.
Despite the problems, it still providesa low-cost and scalable approach to evaluate the generation toxicity of LMs.
Moreover, both ourstudy in Section 7 and Welbl et al.
[15] find that the toxicity scores from Perspective API are stronglycorrelated with human evaluation, thus it is meaningful to approximately measure LM toxicity.
We use the full set of the prompts (around 100k) from R EAL T OXICITY P ROMPT benchmark [10]to evaluate LM generations via Perspective API in terms of Expected Maximum Toxicity and
Toxicity Probability .
Specifically, Expected Maximum Toxicity evaluates the worst-case generation bycalculating the maximum toxicity scores over 25 generations under the same prompt with differentrandom seeds, and averaging the maximum toxicity scores over all prompts.
Toxicity Probabilityestimates the empirical frequency of generating toxic language, which evaluates the probability ofgenerating a toxic continuation (T OXICITY >= 0.5) at least once over 25 generations for all prompts.
We follow Gehman et al.
[10] and restrict the generations up to 20 tokens or below.
We present theautomatic evaluation of five LMs with different parameter sizes in Table 1.
Human Evaluation is indispensable for toxicity evaluation, as toxicity judgments are subjective andshould ultimately be human-centric [15].
Specifically, we adapt the instructions from Welbl et al.[15] and ask human annotators to evaluate the continuations.
More details of human evaluation andhow we ensure the emotional well-being of annotators can be found in Section 7 and Appendix §A.3.
3.2 LM Quality Evaluation
To understand the impact of detoxification, we evaluate the quality of LM along two fronts: perplexityand utility .
Perplexity (PPL) is evaluated on a held-out validation set of pre-training corpus 3 ,which measures both the fluency and coverage of output language.
The utility is estimated by theperformance on downstream tasks.
In particular, we evaluate the accuracy of LMs given 9 differenttasks, covering question answering, natural language understanding, and commonsense reasoning,in the zero-shot learning scheme.
We base the downstream tasks evaluation on Gao et al.
We present the LM quality evaluation of 5 pre-trained LMs in Table 1.
More details about eachdownstream task and the accuracy for each task can be found in Appendix §A.3.
We note some recent work [13, 15] demonstrates that existing detoxification techniques can amplifythe social biases against minority groups.
In this work, we mainly focus on the intrinsic quality of LMand analyze how it degrades after detoxification.
We leave the bias discussion in Appendix §A.10.2.
In the following sections, we use above evaluation protocols to explore the limits of domain-adaptivetraining for detoxification on three dimensions: training corpus, model sizes, and parameter efficiency
[Table]
Impact of Training Corpus
Training corpus is a core factor that impacts the effectiveness and efficiency of domain-adaptivetraining.
The state-of-the-art approach, DAPT [10], adopts a pre-training corpus [17] curated by
Perspective API to construct the training dataset for detoxification.
In this section, we propose
Self-Generation Enabled domain-Adaptive Training (SGEAT), which leverages the generative powerof LM itself to construct a training corpus for domain adaptive training.
To control the variableand have a fair comparison with the existing approach, we also use Perspective API to curate ourself-generated corpus.
We show that SGEAT can further push the limits of domain-adaptive trainingfor detoxification with better data efficiency.
4.1 SGEAT
As shown in Figure 1, SGEAT consists of four steps: 1) prompt construction; 2) self-generation; 3)data filtering; and 4) domain-adaptive training.
Prompt construction is the core part of SGEAT to guide LM to generate a training corpus.
We studythree variants of SGEAT with different prompt designs: 1) SGEAT (standard) uses no prompt andperforms unconditional generation.
2) SGEAT (heuristic) uses a set of manually crafted promptsinspired by the definition of toxicity from Perspective API.
We discuss the set of considered templatesin Appendix §A.4 and report the one that achieves the lowest toxicity in our experiments. 3)
SGEAT (augmented) constructs prompts that tend to yield nontoxic continuations.
Specifically, wefind the most nontoxic documents from the unconditional generation, and split each document intohalf as the prompts and the continuations.
In this way, we obtain the prompts that are highly likely togenerate nontoxic language.
SGEAT (augmented) can also be regarded as a data augmentation of
SGEAT (standard) from the nontoxic distribution.
We present more details in Appendix §A.4.
Self-Generation uses the prompts from the last step to generate up to 1,000 tokens and truncateall the sentences at the end-of-document (EOD) token once generated.
We use nucleus sampling[31] with p = 0 .
9 and the temperature of 1 during generation.
To demonstrate the data efficiency of
SGEAT, we generate only 100k documents in total, in comparison with DAPT in Gehman et al.
[10]that uses 7500k documents from the pre-training corpus.
Data Filtering further filters out toxic samples to ensure the training corpus is mostly nontoxic.
Specifically, we follow the standard DAPT setup in Gehman et al.
[10] and use Perspective APIto annotate the toxicity of the raw generated text.
Different from DAPT that performs aggressivefiltering on pre-training data and only keeps the most nontoxic 2% of the documents, we keep themost nontoxic 50% of the generated text to demonstrate the quality and data efficiency of SGEAT.
We present the curated data toxicity and statistics in Appendix Table 12.
Domain-Adaptive Training leverages the curated nontoxic corpus to further fine-tune the pre-trained
LM with standard log-likelihood loss and adapt it to the nontoxic data domain.
We present moretraining details in Appendix §A.2.
4.2 Evaluation Results of Domain-Adaptive Training
In this subsection, we evaluate existing domain-adaptive training methods on 1.3B LM (similar to
GPT3-XL), and discuss the impacts of model sizes in Section 5.
Baselines: We consider the following domain-adaptive training baselines: DAPT (nontoxic)
[26]uses a nontoxic subset of pre-training corpus annotated by Perspective API to perform domainadaptive training; and Jigsaw (nontoxic) uses a human-annotated nontoxic subset of Jigsaw Toxic
Comment Classification dataset 4 .
We present the evaluation results in Table 2.
Among all domain-adaptive training methods, we findthat SGEAT (augmented) achieves the lowest toxicity scores with moderate perplexity increases andwithout degrading the LM utility accuracy (or even improving).
Specifically, SGEAT (augmented)reduces the toxicity of the standard 1.3B by 0.14 at the cost of a slight PPL increase and does not hurtthe utility of LMs on downstream tasks.
Moreover, we note that although DAPT (nontoxic) uses 3times larger corpus than SGEAT (augmented) (shown in Appendix Table 12), SGEAT (augmented)still achieves lower toxicity than DAPT (nontoxic), which implies that self-generated data has betterdata efficiency for domain-adaptive training.
We think such high data efficiency comes from thefact that i ) the self-generated corpus well captures the high-density regions of the output spaceof a pre-trained LM, and ii ) training on autoregressively generated corpus mitigates the exposurebias [20, 21], which refers to the train-test discrepancy of an autoregressive model.
Thus, whenwe train the LM on the self-generated non-toxic corpus, it tends to increase the likelihood on thenon-toxic density region, which enables data-efficient training to detoxify the model.
The human-annotated nontoxic Jigsaw dataset fails to detoxify the LM and even increases the modeltoxicity.
We speculate the major reason is that the nontoxic subset of the Jigsaw dataset has a muchhigher average data toxicity than SGEAT, as shown in Appendix Table 12.
Among SGEAT methods, we observe that SGEAT (augmented) achieves the best detoxification resultat a similar level of PPL increase, while SGEAT (heuristic) is less effective to detoxify the LM.
Wethink the reason lies in the data diversity:
The unconditional generation covers the diverse regions ofthe generation distribution and yields the most diverse data distribution, and thus SGEAT (standard)also achieves good detoxification performance.
In contrast, SGEAT (heuristic) uses only a singleprompt for generation, which limits the diversity of the generation.
More analysis about promptdesign is in Appendix §A.7.
4.3 Evaluation Results of Decoding-time Methods
Besides the domain-adaptive training baselines, we also compare with decoding-time algorithms:
Word Banning [10] sets the probability of generating any word from a list 5 of profanity, slurs, andswearwords to zero during decoding.
Rejection sampling [15, 13] generates up to K samples giveneach prompt until we obtain a nontoxic sample, otherwise we return the sample with the lowesttoxicity score from Perspective API.
We set K = 4 due to the computational limit.
DE XPERTS [14]is the state-of-the-art decoding-time algorithm for detoxification that uses two auxiliary expert andanti-expert LMs to steer a model’s generation.
When comparing domain-adaptive training methods with decoding-time methods.
We note thatrejection sampling adds 4 × computational overhead during decoding, but is less effective thandomain-adaptive training SGEAT, as LM rarely generates nontoxic continuations given toxic prompts[13].
Although the state-of-the-art DE XPERTS achieves significantly lower toxicity scores than
SGEAT, we also observe that there is a concerning perplexity and utility degradation, with an increaseof 9.47 in PPL and a drop of 9 . 4% in downstream task accuracy.
Such degradation makes thedetoxified 1.3B
LM quality even worse than a standard 126M LM, as shown in Table 1.
We hope thatour findings can motivate researchers to focus more on the trade-off between detoxification and LMquality when designing detoxification algorithms.
Since decoding-time algorithms are orthogonalto domain-adaptive training methods, it is easy to combine both methods together.
Specifically, wereplace the standard 1.3B model used in rejection sampling and DE XPERTS with SGEAT (augmented)detoxified one, and observe that the combined method can yield the lowest toxicity scores amongexisting methods.
Impact of Model Size
We next investigate how the number of model parameters impacts the domain-adaptive training fordetoxification.
Specifically, we show that 1) models with different number of parameters trained onthe same pre-training corpus display similar levels of toxicity; 2) self-generated data consistently
[Table]
demonstrates better detoxification effectiveness than pre-training corpus across different parametersizes; 3) larger LMs require more efforts to reduce the toxicity.
Standard Model Toxicity.
We first evaluate the toxicity of 5 standard LMs across different parametersizes in Table 1 and Table 8.
We observe that the standard LMs, pre-trained on the same pre-trainingdata with different parameter sizes, display similar levels of toxicity.
It suggests that the toxicitycomes from the dataset, instead of the model size .
Detoxification Effectiveness of SGEAT.
We then evaluate our best SGEAT (augmented) and comparewith the best domain-adaptive training baseline DAPT (nontoxic) in Table 3.
We note that SGEATconsistently outperforms DAPT over different sizes even when using 1/3 smaller training corpus.
Forexample, SGEAT (augmented) can reduce the toxicity probability from 57% to 30% for the 126M
LM, 7% lower than DAPT.
These results confirm that: the self-generated corpus is more efficient todetoxify the LM than using the curated corpus of pre-training data.
Larger-scale LMs requires more endeavors to detoxify.
From Table 3, we observe the detoxificationeffectiveness decays for both DAPT and SGEAT with the increase of LM parameter sizes.
For instance,the toxicity probability of the 530B SGEAT LM is only the 16% lower than the standard 530B LM,compared to the drop of 27% toxicity probability for the 126M one.
We figure the potential reasonof such small improvement on larger LM is that large LM tends to require more training data andfine-tuning epochs to detoxify.
Therefore, we conduct additional experiments on the 530B LM, byeither increasing the training epochs from 3 to 5 or generate more data from 50k to 100k samplesfor adaptive training.
We find that while both methods further reduce the toxicity of the 530B LM,training for more epochs might lead to model overfitting and hurts the PPL and downstream accuracyby a large margin.
In contrast, training with more data demonstrates a better trade-off betweendetoxification and LM quality.
It implies that it needs more endeavors to detoxify large-scale LMs.1 epoch 3 epoch 5 epoch
LM Quality Evaluation.
We also evaluate whetherdomain-adaptive training impacts the perplexity andutility of LMs in Table 3.
When trained within 3epochs, we find that the PPL of LMs slightly increases and the LM utility drops a little in mostcases, which suggest that models gradually adaptto the nontoxic domain without a significant sign ofoverfitting or degradation in terms of LM quality.
Domain Adaptation v.s. Overfitting.
We visualizethe trade-off at different training phases in Figure2 for 530B LM.
Specifically, we record the validation perplexity and model toxicity after 1, 3, and 5training epochs for DAPT and SGEAT.
We observethat at the beginning of training, the model toxicitydrops substantially and barely sacrifices the model
PPL (steep slope).
Then it is gradually adapted towards the nontoxic domain.
SGEAT demonstrates a [Table] [Table] better trade-off between toxicity and quality, as SGEAT achieves substantially lower toxicity with thesame PPL.
Finally, we observe the curve is becoming more flat, especially for DAPT, which indicatesthe transition from the domain adaptation to overfitting.
For LMs with different sizes fine-tuned with different methods, we find 3 epochs is a good cut-offpoint for whole model adaption, which achieves good trade-off between model toxicity and perplexity.
This rule of thumb is also aligned with previous study [10].
Parameter-efficient Training
To cope with the challenges of large-scale LMs, we explore two parameter-efficient training paradigms:adapter [18] and prefix tuning [19], and evaluate whether they can improve the LM quality andachieve a better trade-off between detoxification and LM quality than whole model adaption.
Weshow that: in the scenario of detoxification, 1) adapter demonstrates a better trade-off than prefixtuning, and 2) adapter can further mitigate the drop of LM quality and improve the trade-off uponwhole-model adaptation for large-scale LMs.
6.1 Comparison between Adapter and Prefix Tuning
Both adapter and prefix tuning add additional parameters to the standard LM, and only optimize theadded parameters during training without perturbing the original LM parameters.
Such paradigmprovides the flexibility, especially for large-scale LMs, to adapt to different domains with a fewadditional parameters, rather than heavily fine-tune the whole model with multiple copies of thewhole model parameters for different domains.
In this study, we further investigate whether suchtraining schemes can provide more advantages to detoxify LMs.
Adapter [18] adds additional bottleneck projection layers to each transformer layer with residualconnections.
At the beginning of the training, the projection layer is initialized to almost zero toimprove the training stability.
Prefix tuning [19] appends additional continuous “prefix” vectors tothe input to better steer LMs’ generations.
To have a comprehensive understanding and comparisonbetween adapter and prefix tuning, we first perform ablation studies on small-scale 1.3B LM overthe key hyper-parameters: the projection size for adapter and the prefix length for prefix tuning.
Wefollow the same training schedules as whole model adaptation but train more epochs so that the PPLreaches a similar level as whole model adaptation.
We present the evaluation results in Table 4.
When comparing Table 4a with Table 4b, we observe that adapter demonstrates a better trade-offbetween detoxification and LM quality than prefix tuning.
We figure the possible reasons are twofolds: 1) given the same projection size and prefix length, the number of additional parameters ofadapter is around twice more than prefix tuning, which gives more capacity for adapter to performdomain adaptation; 2) however, while longer prefix length could give more capacity to steer themodel generation, it also adds too many irrelevant contexts, which not only hurts the perplexity of the
LM but also slows down the decoding speed.
Compared to the whole model adaption, adapter doesnot show significant advantages in terms of detoxification and LM quality for small-scale models like1.3B one.
For adapter results with different projection sizes, we observe that a larger projection size yields better detoxification effectiveness possibly due to larger model capacity.
We thus apply adapterwith the projection size = 1024 to larger-scale LMs (8.3B and 530B) and investigate whether it cansolve the challenges of large-scale LMs.
6.2 Apply Adapter to larger-scale Models
We follow the same training schedules as the whole model adaptation to train the adapters for largerscale LMs.
We stop training when they reach similar levels of toxicity as the whole model adaptation,and evaluate the perplexity and utility of LMs in Table 5.
We can see that for larger-scale LMs,adapter can not only improve the parameter efficiency, but also mitigate the PPL and the LM qualitydrop.
In particular, for the 530B model, adapter can mitigate the drop of PPL for at most 0.64 andimprove the average downstream task accuracy by 0 .
Human Evaluation
We further verify our findings via human evaluation on the standard models, DAPT, SGEAT, anddecoding-time algorithm DE XPERTS across five LM sizes.
[Figure]
Setup.
We sample the 300 prompts from R EAL T OXICITY P ROMPT benchmark while keeping the ratio of toxic and nontoxic prompts to1:3 as the same as the full set, and evaluatethe continuations of each model.
We follow
Welbl et al.
[15] to ask LMs to generate upto 100 tokens and avoid incomplete sentencesand collect the most toxic continuations via
Perspective API over 25 generations.
Finally,we gather 5,700 continuations from 19 modelsand randomly shuffle them for human evaluation.
Then we group samples into a batch of10, and assign them to 5 annotators.
In total187 workers from Amazon MTurk participatedin the evaluation.
To consider the annotators’well-being, we make sure the average numberof toxic samples (T OXICITY >= 0.5 evaluated by Perspective API) is less than or equal to 3 in eachbatch of 10 samples.
To calculate the average scores of annotations, we follow Welbl et al.
[15] tomap “Very Toxicity” and “Toxic” to 1 , “Not Toxic” to 0 , and discard “Not Sure“ annotations.
We average the scores from 5 annotators for each sample and then report the averaged number overthe 300 prompts in Figure 3.
The detailed scores can be found in Table 7 in Appendix.
We presentmore details in Appendix §A.3.
By comparing the objective evaluation with human evaluation,we observe that the toxicity scores from the human evaluation are mostly aligned with objectiveevaluation via Perspective API.
Such findings are also confirmed by Welbl et al.
The humanevaluation also verifies that i ) LMs of different sizes have similar levels of toxicity, and ii ) LMs oflarger sizes present more challenges to detoxify.
Conclusion
In this work, we explore the limits of domain-adaptive training for detoxifying language along three aspects: 1) training corpus; 2) model size and 3) parameter-efficient training.
We argue that existing approaches either suffer from limited detoxification effectiveness or significantly sacrifice the language model quality to detoxify LMs.
We propose Self-Generation Enabled domain-Adaptive
Training (SGEAT) which leverages the generative power of LMs for data-efficient and effectivedetoxification.
We comprehensively study detoxifying LMs with parameters sizes ranging from
M up to 530B and find interesting properties of large-scale LMs.
We demonstrate that adapter provides parameter-efficient training and achieves a better trade-off of toxicity and LM quality.
We hope our work can shed light on the development of detoxification techniques that can largely reduce toxicity while maintaining good perplexity and downstream task accuracies
Checklist
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix Section A.10.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Our work focus reducing the toxicity of LMs. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No](b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix Table 10 and
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[Yes] See Appendix A.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable? [Yes]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[Yes] See Appendix A.
[Table]
A AppendixA.1 Details of Pre-trained LMs
The architecture details of pre-trained LMs are in Table 6.
The corresponding perplexity anddownstream task accuracy is shown in Table 8.
[Table]
A.2 Training Details
Throughout our experiments, we follow the setup of Gehman et al.
[10] and fine-tune the LMs for 3epochs.
We use the batch size equal to 512 with sequence length 2048.
We use the Adam optimizer(epsilon=1e-5, beta-1=0.9, beta-2=0.95) with initial lr = 1e-5 for 530B LM, while we use lr = 2e-5 forall other LMs.
We set weight decay to 0.1 for all LMs.
Our experiments are conducted on the DGX
A100 servers with 8x A100 GPUs.
A.3 Details of Evaluation Protocols
Details of Human evaluation Asking humans to annotate toxicity necessarily exposes them totoxic language.
Before conducting our study, it was reviewed by our Institutional Review Board(IRB).
Participants were recruited through Amazon MTurk.
Since toxicity evaluation is subjective todifferent social groups, we do not have any constraints on the demographic background of annotators.
Since our toxicity evaluation mainly focuses on English, we also constrain the regions of annotatorsto the United States, Canada, Australia, and the United Kingdom.
To improve the quality of theannotations, we require the participated annotators to have at least 1000 approved HITs and a lifelong
HIT approval rate is greater than 98% .
We group continuations in a batch of 10 samples and assign them to annotators.
The payment rate foreach batch of samples is $0 .
We follow the definition of Perspective API and Welbl et al.
[15] on toxicity as follows:
• Very Toxic: Text that is very hateful, aggressive, disrespectful, or otherwise very likely to make a
user leave a discussion or give up on sharing their perspective.
• Toxic: Text that is rude, disrespectful, unreasonable, or otherwise likely to make a user leave a
discussion or give up on sharing their perspective.
• Not Toxic: Neutral, civil, or even nice text very unlikely to discourage the conversation.
We adapt the instructions from Welbl et al.
[15] and show the annotation instructions on Amazon
MTurk platform in Figure 4 and Figure 5.
We put the detailed human-annotated toxicity scores in Table 7 and compare with Perspective API.
We observe that the toxicity scores from the human evaluation are mostly aligned with objectiveevaluation via Perspective API.
Details of PPL Evaluation We evaluate the LM PPL on a held-out validation set from the pretraining corpus.
Note that, the validation set can be different from the one in Smith et al.
[6] due todifferent random seed and chunking.
Details of Downstream Task Evaluation We consider the following 9 downstream tasks:
• ANLI [32] is a large-scale NLI adversarial benchmark dataset.
• BoolQ [33] is a question answering dataset for yes/no questions.
• Hellaswag [34] is a commonsense NLI dataset.
• LAMBADA [35] is a cloze test (word prediction) dataset.
• PIQA [36] is a physical commonsense reasoning and a corresponding benchmark dataset.
• RACE [37] is a large-scale reading comprehension dataset.
• WiC [38] is a multilingual Word-in-Context Dataset for the evaluation of context-sensitive word
embeddings.
• WinoGrande [39] is commonsense reasoning for pronoun resolution problems.
Our evaluation code is based on Gao et al.
[Table] [Table]
A.4 Details of Prompt Design
Our prompt exploration starts from unconditional generation, and then moves on to the conditionalgeneration scenarios for investigating which prompts can best facilitate LMs to generate high-qualitynontoxic data.
A.4.1 Unconditional Generation
By only taking the start-of-sentence token 6 as input, unconditional generation unveils the intrinsicself-generation quality given no prompt and thus tends to yield the most diverse data distribution.
During the generation, we use different random seeds to collect a total of 100,000 self-generatedsamples as SGEAT (standard).
A.4.2 Conditional Generation
Prompts provide language models with context to generate a variety of desired contents.
Fordetoxification purposes, we aim to design prompts such that language models can generate highquality and diverse nontoxic data.
Manually-crafted Prompts (SGEAT (heuristic))
We start with the definition of toxicity following
Perspective API as the seed template.
Then we manually craft prompts to perform self-generationand evaluate their detoxification performances.
Specifically, we follow the definition of toxicity from Perspective API as well as the inspiration by therecent work from Prompt Engineering [40, 41, 19, 42–45] that repeating the prompts and prompting
LMs in the format of Question Answering, and design the prompts below to study the generation andunderstanding power of the LMs,
Negative Prompt (for once).
The following text does not contain rude, disrespectful and unreasonable languages: ___.
Positive Prompt (for once).
The following text contains kind, respectful, and reasonablelanguages: ___.3.
Positive Prompt twice.
Repeating the above prompt twice.4.
Positive Prompt in the format of QA.
Question :
The following text contains kind, respectful,and reasonable languages.
Answer : ___.5.
Positive Prompt twice in the format of QA.
Repeating the above prompt twice.
For each prompt, we generate 100,000 continuations, and use them as a self-generated dataset beforedata filtering.
Automatically-constructed Prompts (SGEAT (augmented))
As we will show in the experiment,manually-crafted prompts suffer from the limited templates and may impact the diversity of theself-generated data.
Inspired by Gehman et al.
[10], we design the following process to automaticallyconstruct prompts: i)
We use the unconditional generated data in §A.4.1 as the source data to constructprompts, as it represents the distribution of LM while keeping the data diversity.
ii)
We apply
Perspective API to annotate the unconditionally generated data, and select the most nontoxic 25%data, which in total collects the most nontoxic 25,000 documents.
iii)
We split each document intotwo parts, yielding 25,000 prompts and continuations.
For each prompt, we generate 4 continuationsand eventually collect 100,000 self-generated samples before data filtering.
We list a glossary of prompts (automatically-constructed prompts and manually-crafted prompts) weconsidered along with their explanation and examples in Table 9.
A.5 Unprompted Toxicity
During unconditional generation to construct prompts, we can also evaluate the unprompted toxicity(i.e.
, unconditional generation scenario) of LMs of different sizes.
We present the unconditionalgenerated data toxicity in Table 11.
We can see that standard LMs yield similar levels of data toxicity in the unconditional generationscenario, which is also aligned with the observations in our main experiment.
A.6 Data Toxicity given Automatically-constructed Prompts
Furthermore, given our automatically-constructed prompts, our SGEAT demonstrates lower datatoxicity than an unconditional generation, as shown in Table 11.
A.7 Benchmark and Analysis of Prompt Design
As the core of SGEAT is the prompt design, we perform a systematic study on the 1.3B LM toevaluate how different prompts impact the self-generated data quality, which further affects thedetoxification performance. We evaluate the prompts following two fronts: i ) Data Toxicity , whichdirectly evaluates the generated data toxicity scores via Perspective API in Table 12.
Specifically, wereport the average toxicity of the generated data, the probability of generating toxic and nontoxic
[Table]
Examples
[Table]
The following text does not contain rude, disrespectful, and unreasonable languages: ___________________________________.
The following text does not contain rude, disrespectful, and unreasonable languages: Black Mountain College in North Carolina awarded the prize last year to a team that used algorithms to predict whether a ball would turn into a goal.
A world record this time will be considered statistically significant.
The team, from the
Netherlands, started by setting out the locations of six ball goals in three different games: grass (A); turf (B); and cobblestone (C)...
( Explanation : The prompt is expressed using negation “does not”.)
[Table]
The following text contains kind, respectful, and reasonable languages: ___________________________________.
The following text contains kind, respectful, and reasonable languages: (0 / 1) Review score is calculated from customers who have booked through our site.
Add Your Review This review is the subjective opinion of a TripAdvisor member and not of TripAdvisor LLC.
The hotel itself is a fairly large and nicely maintained property, the rooms are a little dated but comfortable, with clean and useful toiletries (there’s a decent sized vanity), well-appointed bathrooms, and each room has a fridge and tea/coffee maker..
( Explanation : The prompt is expressed via the positive tone.)
[Table]
The following text contains kind, respectful, and reasonable languages: The following text contains kind, respectful, and reasonable languages: ___________________________________.
The following text contains kind, respectful, and reasonable languages: The following text contains kind, respectful, and reasonable languages:
These articles discuss the following topics: 1.
the 44th Annual Meeting of the American Association of Applied Science.
October 28-30, 2012 in the Omni Shoreham
[Table]
[Table]
[Table]
samples, their corresponding toxicity, and their toxicity scores after filtering; and ii ) Model Toxicity ,which evaluates the final performance fine-tuned with the generated data in Table 10.
Analyzing both Table 10 and 12, we have the following observations: i)
Using all automaticallyconstructed prompts provides the best toxicity reduction performance among all the prompt designs.
This result is also aligned with the observation in Table 12 that automatically-constructed promptsyield the least average data toxicity (0.07).ii)
Low data toxicity does not necessarily lead to good model toxicity after fine-tuning.
Diversity alsomatters.
When we choose the least nontoxic prompt from automatically-constructed prompts as thesingle prompt for generation, we find that although the generated dataset achieves the average datatoxicity as low as 4e-4, the toxicity reduction is not as effective as using all automatic-constructedprompts.
We think the reason is that both data toxicity and data diversity contribute to the detoxification effectiveness.
The prompts with lower data toxicity can more effectively pull the generationdistribution from the toxic domain to the nontoxic domain, while the higher prompt diversity cancover more regions of the generation distribution, thus yielding lower model toxicity.iii) Manually-crafted prompts are not enough to generate high-quality non-toxic data.
Therefore,manually-crafted prompts yield worse detoxification effectiveness than unconditional generation.
The unconditional generation covers the diverse regions of the generation distribution and yields themost diverse data distribution, and thus also achieves good detoxification performance.
In contrast,human-crafted prompts use only a single prompt for generation, which limits the diversity of thegeneration.
Moreover, the generation tends to follow the topics of the prompts related to toxicity, and [Table] [Table] thus is more likely to yield toxic samples than unconditional generation, as shown in Table 12.
Wealso note that repeating the positive prompt twice can cause lower toxicity in the continuations, whileprompting the language model in the question-answering format [46] is less helpful for generatinglower toxicity data.
In addition, using negative prompts may even backfire and increase the modeltoxicity, suggesting that it is better to prompt language models in a positive way instead of using negations.
iv) Human-annotated nontoxic Jigsaw dataset fails to detoxify the LM, and even increases the model
toxicity.
We think there are two main reasons: 1) the nontoxic subset of the Jigsaw dataset has much
higher data toxicity than the filtered OWTC; 2) the Jigsaw data has some domain shift from thepre-training data distribution
and thus limits the effectiveness for detoxification
A.8 Downstream Task Accuracy
We present the detailed downstream task accuracy of each method for nine tasks in Table 13, 15, 14,and 16.
A.9 Perplexity Evaluation on Nontoxic Validation Set
Hypothesis We hypothesize that the reasons for the perplexity increase on the validation set of thepre-training data after domain-adaptive training are two fold: 1)
The validation set may contain toxiclanguage, while the LMs are already adapted to the nontoxic domain.
Thus it is expected that the
LM loss on the toxic portion increase after detoxification, which leads to the PPL increase on thefull validation set.
The filtered non-toxic corpus are not perfect (e.g., poor coverage of languagefor different topics), which may hurt the LM‘s quality after domain-adaptive training.
This is alsoconfirmed by the degradation of down-stream task accuracy.
To verify the hypothesis, we further filter our validation set based on Perspective API to constructseveral nontoxic corpora, and evaluate the LM PPL on these nontoxic corpus. [Table] [Table]
Setup We construct three validation set with different filter rates as shown in Table 17, where
Nontoxic @
x% refers that we keep the most x% of nontoxic documents for PPL evaluation.
We alsopresent the PPL evaluation on Nontoxic @ 10% for all detoxification methods we consider for the1.3B model in Table 18.
Analysis We find that: 1)
The PPL increase on the nontoxic subsets of validation corpus is less thanthat on the full validation set.
This suggests that the toxic documents in the validation set indeed leadto some of the PPL increase for our detoxified language models.
The lower the average toxicityscore the validation set has, the less PPL increases.
The trend of PPL increase on nontoxic corpusis almost the same as that on the full validation set.
Thus we report the standard PPL increase on ourfull held-out set in our main paper to reflect the level of LM quality degradation.
[-The results are shown in Table
-]
A.10 Discussion and Case Studies
A.10.1 Error Analysis of Perspective API
Although Perspective API and human annotation are statistically highly aligned in most cases asconfirmed in our §7 and recent work[15], we also note that in some cases, Perspective API canalso make mistakes.
For example, Perspective API can overestimate the toxicity of test samples.[15] find that the nontoxic continuations from detoxified models verified by human annotators aresometimes classified as highly toxic by Perspective API.
We also confirm the same phenomena in ourexperiments.
Quantitatively, we find that among all toxic (classified by Perspective API) samples generated by
DE XPERTS + SGEAT, around 34% samples are actually annotated as nontoxic by the majority voteof 5 human judges.
Qualitatively, we show a qualitative example of failure cases of Perspective API below: [Table] [Table] [Table]
“Let us help you with your cleaning!
Spend your free time doing what you want to do, not what you feel you need to do.
Established in 1987, SelClene Herts pride themselves on supplying reliable domestic cleaners thatyou can trust.
The reputation of SelClene (St Albans) Ltd has been built up over the years by our manyhappy customers across Hertfordshire and Buckinghamshire.
We have a good supply of domesticcleaners across the area to do your housework and keep your home spick and span.
Our customers across Herts & Bucks use our cleaning services because of our expertise and customerservice.”
The above paragraph comes from the pre-training corpus and it is surprisingly the most “toxic”document classified by Perspective API with toxicity score equal to 0 .
While we have submittedthe error case to Perspective API, we hope our findings can motivate the development of more robustand unbiased hate speech detection model.
A.10.2 Bias against Marginalized Groups
We follow the setting of Welbl et al.
[15] and evaluate the PPL of the 1.3B standard LM and
SGEAT (augmented) fine-tuned LM on the [Table] [47] as shown in Table 19.
The former contains Wikipedia sentences about female and male actors,and the latter domain contains sentences about people with different ethnic backgrounds [15].
Wefind that: ( i ) LM PPL increases moderately on the BOLD dataset after effective detoxification, whichis aligned with our findings in §4.2.
( ii )
There is no noticeable discrepancy of PPL increase amongmale and female in the gender domain, which suggests that SGEAT does not exacerbate the genderbiases.
( iii )
There is a higher PPL increase for the Hispanic group than other demographic groupsin the ethnicity domain.
We hypothesize that such bias mainly comes from the pre-training modeland corpus, because the pre-trained Standard model already has much higher perplexity for Hispanicgroup.
We leave it as an important future direction to mitigate the social biases of pre-trainedfoundation models.
A.10.3 Limitation of SGEAT
Although we observe that SGEAT has demonstrated the best detoxification effectiveness and the besttrade-off between toxicity and PPL, we note that there is still a lot SGEAT can improve on.
First, we acknowledge that similar to DAPT, SGEAT also relies on Perspective API.
Since Perspective
API is imperfect and is known to amplify the social bias against different demographic groups, SGEATmay also impacted due to the use of Perspective API for filtering.
However, we believe that SGEATcan also get more benefits with a more robust, unbiased, and fair hate speech detector.
Second, in this paper, we mainly focus on the intrinsic quality of LMs and analyze the trade-offbetween toxicity and quality.
While recent work demonstrates that detoxification methods mayamplify social biases, we leave it as a future work to analyze the bias impact after detoxification.
A.10.4 Case Studies of SGEAT
Warning!!
The following content contains toxic language.
Now, we present both successful and failure cases of detoxification using SGEAT (augmented)
for530B LM in Table 20 and Table 21.
[Table]
when he arrived in
[Table]
of
[Table]
Periodico in
[Table]
He
[Table]
editor, Antonio Cano.
The protesters, some of whom wore Spanish
[Table]
, shouted
[Table]
" at Puigdemont.
Spanish prosecutors have asked a judge to issue a warrant for the
[Table]
former
Catalan president Carles Puigdemont and other members of
[Table]
and to
[Table]
[Table]