Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
Abstract
Pre-trained language models (LMs) are shown to easily generate toxic language.
In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models.
We conduct this study on three dimensions: training corpus, model size, and parameter efficiency.
For the training corpus, we demonstrate that using self-generated datasets consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 13 smaller training corpus.
We then comprehensively study detoxifying
LMs with parameter sizes ranging from 126M up to 530B (3 × larger than GPT3), a scale that has never been studied before.
We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to unlearn the toxic content seen at pretraining.
We also explore parameter-efficient training methods for detoxification.
We demonstrate that adding and training adapter -only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale models.
Our code will be available at: https://github.com/NVIDIA/Megatron-LM/ .
Introduction
Large-scale pre-trained language models (LMs) [1–6] have demonstrated substantial performance gains on various NLP tasks, especially when scaling up the sizes of models.
However, recent studies [7, 8] show that generative LMs can generate toxic and biased language, which raises ethical concerns for their safe deployment in real-world applications.
Previous methods on reducing the toxicity of LMs can be categorized as: decoding-time methods, pre-training-based methods and domain-adaptive training methods.
Decoding-time methods [9–14] manipulate the output distribution or input prompts at the inference stage without modifying the original model parameters.
These methods can be flexible, but they either resort to some simple word filtering strategies [10], or increase the computational cost at the inference stage.
For example,
PPLM [9] requires multiple iterations of backward propagation through the LM when generating every token, which makes it prohibitively expensive to be deployed to production especially for large-scale LMs.
In contrast, pre-training-based methods directly filter out the potentially toxic content within the pre-training corpus and retrain the model from scratch [e.g., 15].
However, it is difficult to determine the filtering criterion beforehand, and pre-training a large LM multiple times from scratch is quite expensive.
Domain-adaptive training methods [10, 16] further fine-tune the pre-trained LMs on carefully curated datasets (e.g., Jigsaw, filtered OWTC [17]).
For instance, Gehman et al.
[10] construct a nontoxic data corpus from an existing dataset, OWTC, via the Perspective API 4 and perform the fine-tuning on the nontoxic corpus.
Domain-adaptive training is more flexible than pre-training methods, as one can still customize the model after the expensive pre-training process.
Compared to the decoding-time methods, domain-adaptive training methods have the following advantages: i ) they can achieve fast and memory-efficient inference, thus can be deployed in broader systems; and ii ) they can largely reduce the model toxicity while still maintaining good LM quality measured by perplexity and downstream task performance as we will show in this work.
In this paper, we explore the limits of domain-adaptive training for detoxifying language models along the following three aspects: 1 ) Training Corpus : Unlike previous methods using curated pre-training corpus for detoxification, we propose to leverage the generative power of LMs to generate nontoxic corpus, which achieves better data efficiency for detoxification.
2 ) Model Size : We systematically study and mitigate the toxicity issues in LMs with parameter sizes ranging from 126M to 530B, a scale that has never been studied before in this domain.
3 ) Parameter-efficient Training : We investigate two parameter-efficient paradigm: adapter [18] and prefix-tuning [19], and compare them with whole model adaptation in a systematic way.
We hope our work can shed light on the challenges of detoxifying large-scale LMs, as well as motivate the development of detoxification techniques that are effective and parameter-efficient without significantly hurting the LM quality.
Summary of Contributions:
• We identify the trade-off between detoxification effectiveness (measured by Perspective API and human evaluation) and language model quality (measured by validation perplexity and downstream task accuracy).
Existing approaches either suffer from limited detoxification effectiveness or significantly sacrifice the language model quality to detoxify generative LMs.
• We propose Self-Generation Enabled domain-Adaptive Training (SGEAT) that uses a selfgenerated dataset for detoxification.
It mitigates the exposure bias [20, 21] from the discrepancy between teacher-forced domain-adaptive training and autoregressive generation at test time, and thus achieves better data efficiency.
In particular, we demonstrate that it consistently outperforms the baseline approach with domain-adaptive training on pre-training data (DAPT) by a wide margin across various model sizes in terms of automatic and human evaluations, even when we use only a 13 smaller corpus for training.
By combining SGEAT with the state-of-the-art decoding-time method, we can further reduce the toxicity of large-scale generative LM.
• From the perspective of model size, we find that: i)
Large LMs have similar toxicity levels as smaller ones given the same pre-training corpus.
This implies the toxicity comes from the training dataset, instead of the model size.
ii)
Large LMs require more efforts (e
g., larger training corpus) to reduce toxicity.
• We explore two parameter-efficient training methods for detoxification, and observe that: i) domain-adaptive training with adapter achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale LMs, and the improvement is more significant when the size of LMs increases; ii) prefix-tuning is less suitable for detoxification and demonstrates limited detoxification effectiveness and perplexity control.
We organize the rest of the paper as follows.
We discuss related work in § 2 and present our evaluation protocols in § 3.
We then systematically explore the domain-adaptive training with respect to training corpus in § 4, model sizes in § 5, and parameter efficiency in § 6.
We present the human evaluation result in § 7, discuss the relationship between toxicity and bias in § 8.1, and conclude the paper in § 9.
Some text samples can be found in Appendix D.
Related Work
Large-scale language models (LM) have achieved state-of-the-art performance on various downstream tasks.
However, they also exhibit undesirable behaviors in terms of robustness, privacy, and ethical issues [10, 22–25]. For example, since they are pre-trained over a sizable collection of online data,
[Table]
they are unavoidably exposed to certain toxic content from the Internet. Recent studies [e.g., 26– 28] show that pre-trained masked LMs display different levels toxicity and social biases.
Another line of work focuses on the toxicity of autoregressive LMs.
For instance, Wallace et al.
[8] first demonstrate that synthetic text prompts can cause racist continuations with GPT-2.
Gehman et al.
[10] extend the analysis of LM toxicity to non-synthetic prompts
[Table]
and create a benchmark dataset
R [Table] P ROMPTS to provide a standard evaluation protocol via Perspective API to measure
LM’s toxicity, which is adopted by many previous work.
In this paper, we follow the standard setting to compare different detoxification approaches on different-sized LMs.
Decoding-time methods They manipulate the decoding-time behavior of the LMs without changing the model parameters [9–14].
Simple approaches such as word filtering and vocabulary shifting [10] directly lower the probability of toxic words (e.g., swearwords, slurs, vulgar slang) being generated.
Though efficient, such approaches fail to consider the semantic meaning of the generated text at the sequence level.
Thus, it cannot completely prevent from generating toxic sentences which contain no undesirable words from the blocklist [15] ( e.g., “ poor people don’t deserve to live in nice houses ”).
Xu et al.
[13] perform sentence-level filtering by generating K continuations given the same prompt and returning the most nontoxic sentence.
Similarly, Self-Debiasing [11] uses K manually crafted templates to manipulate the decoding probability distribution and dynamically set the probability of toxic words to be low.
However, these methods lead to K times longer than the normal decoding.
PPLM [9] iteratively adds perturbation on the context vector at each step of decoding.
Though with better detoxification effectiveness, it suffers much more computational overhead due to multiple iterations of forwarding and backward propagation to generate the perturbations.
GeDi [12] guides generation at each step with a second LM trained on nontoxic data by computing classification probabilities for all possible next tokens.
However, it requires an external LM trained on non-toxic data, which is not easy to access in practice.
DE XPERT [14] controls the generation of large-scale pre-trained LM with an “expert” LM trained on non-toxic data and “anti-expert” LM trained on toxic data in a product of experts [29].
It achieves the state-of-the-art detoxification results on
R EAL T OXICITY P ROMPTS , but sacrifices the validation perplexity and downstream task accuracy.
Domain-adaptive training methods They fine-tune the pre-trained LMs to the non-toxic domain by training on curated nontoxic data [10, 16, 30].
Gehman et al.
[10] use the DAPT framework [30] to further train LMs on the nontoxic subset (filtered via the Perspective API) of pre-training corpus, OWTC, with GPT-2.
Besides DAPT, Gehman et al.
[10] propose to fine-tune on a corpus with toxicity attribute token and prepend the nontoxic attribute token as prompt to yield nontoxic generation.
Solaiman and Dennison [16] propose a human-crafted Values-Targeted Datasets to change model behavior and reflect a set of targeted values.
Baheti et al.
[31] focus on mitigating the offensive behavior in dialogue systems.
They leverage crowd-sourcing to label a conversation dataset generated by an existing dialogue model, and use it for offensive detection and mitigating the offensive behavior via the controlled text generation.
In this work, we focus on exploring the limits of domain-adaptive training methods to reduce the toxicity of language models, while maintaining good validation perplexity and downstream task accuracy.
Reinforcement learning (RL) methods There are two concurrent work [32, 33] that study the toxcity behavior of LM with RL.
InstructGPT [32] requires collecting human demonstrations and rankings of model outputs for two-stage fine-tunings.
It generates 25% fewer toxic outputs with respectful instruction on R EAL T OXICITY P ROMPTS than 175B GPT-3.
In contrast, our SGEAT reduces 27% toxic outputs from 530B model on R EAL T OXICITY P ROMPTS , and the improvements are higher for smaller models (e.g., reduces 37% toxic outputs from 8B model).
To identify the toxic
LM behavior, Perez et al.
[33] uses RL to improve the generation of adversarial test cases.
[Figure]
Evaluation Protocols
In this section, we present our principle for evaluating different detoxification methods.
Specifically, we emphasize that detoxification method should focus on both reducing the model toxicity and maintaining the model quality after detoxification.
We first discuss the protocol for LM toxicity evaluation, and then present the protocol to evaluate the LM quality before and after detoxification.
Pre-trained LMs.
We investigate the toxicity of a variety of standard GPT-3 like LMs with different parameter sizes, ranging from 126M (similar to GPT-3 Small), 357M (similar to GPT-3 Medium), 1.3B (similar to GPT-3 XL), 8.3B to the largest 530B [6].
All of the models are based on Transformer [34] with different hidden dimension, number of layers, and attention heads.
We present more details in Appendix §A.1.
All standard models are pre-trained on the same pre-training corpus, which is an English text corpus constructed from 15 high-quality datasets.
3.1 Toxicity Evaluation
In this work, we follow prior work [15, 10] and perform both automatic evaluation and human evaluation to measure an LM’s tendency to generate toxic language.
Automatic Evaluation relies on Perspective API, an online automated model for toxic language and hate speech detection.
As discussed in the recent work [13, 15, 10], such a model is imperfect and demonstrates biases against different demographic groups.
Despite the problems, it still provides a low-cost and scalable approach to evaluate the generation toxicity of LMs.
Moreover, both our study in Section 7 and Welbl et al.
[15] find that the toxicity scores from Perspective API are strongly correlated with human evaluation, thus it is meaningful to approximately measure LM toxicity.
We note that Perspective API update the models regularly.
The scores returned by Perspective API may change over time.
The toxicity scores reported in the following sections were evaluated before May 2022.
We use the full set of the prompts (around 100k) from R EAL T OXICITY P ROMPT benchmark [10] to evaluate LM generations via Perspective API in terms of Expected Maximum Toxicity and
Toxicity Probability .
Specifically, Expected Maximum Toxicity evaluates the worst-case generation by calculating the maximum toxicity scores over 25 generations under the same prompt with different random seeds, and averaging the maximum toxicity scores over all prompts.
Toxicity Probability estimates the empirical frequency of generating toxic language, which evaluates the probability of generating a toxic continuation (T OXICITY >= 0.5) at least once over 25 generations for all prompts.
We follow Gehman et al.
[10] and restrict the generations up to 20 tokens or below.
We present the automatic evaluation of five LMs with different parameter sizes in Table 1.
Human Evaluation is indispensable for toxicity evaluation, as toxicity judgments are subjective and should ultimately be human-centric [15].
Specifically, we adapt the instructions from Welbl et al.
[15] and ask human annotators to evaluate the continuations.
More details of human evaluation and how we ensure the emotional well-being of annotators can be found in Section 7 and Appendix §A.3.
3.2 LM Quality Evaluation
To understand the impact of detoxification, we evaluate the quality of LM along two fronts: perplexity and utility .
Perplexity (PPL) is evaluated on a held-out validation set of pre-training corpus 5 ,
[Table]
which measures both the fluency and coverage of output language.
The utility is estimated by the performance on downstream tasks.
In particular, we evaluate the accuracy of LMs given 9 different tasks, covering question answering, natural language understanding, and commonsense reasoning, in the zero-shot learning scheme.
We base the downstream tasks evaluation on Gao et al.
We present the LM quality evaluation of 5 pre-trained LMs in Table 1.
More details about each downstream task and the accuracy for each task can be found in Appendix §A.3.
We note some recent work [13, 15] demonstrates that existing detoxification techniques can amplify the social biases against minority groups.
In this work, we mainly focus on the intrinsic quality of
LM and analyze how it degrades after detoxification.
We leave the bias discussion in §8.1.
In the following sections, we use above evaluation protocols to explore the limits of domain-adaptive training for detoxification on three dimensions: training corpus, model sizes, and parameter efficiency.
Impact of Training Corpus
Training corpus is a core factor that impacts the effectiveness and efficiency of domain-adaptive training.
The state-of-the-art approach, DAPT [10], adopts a pre-training corpus [17] curated by
Perspective API to construct the training dataset for detoxification.
In this section, we propose
Self-Generation Enabled domain-Adaptive Training (SGEAT), which leverages the generative power of LM itself to construct a training corpus for domain adaptive training.
To control the variable and have a fair comparison with the existing approach, we also use Perspective API to curate our self-generated corpus.
We show that SGEAT can further push the limits of domain-adaptive training for detoxification with better data efficiency.
4.1 SGEAT
As shown in Figure 1, SGEAT consists of four steps: 1) prompt construction; 2) self-generation; 3) data filtering; and 4) domain-adaptive training.
Prompt construction is the core part of SGEAT to guide LM to generate a training corpus.
We study three variants of SGEAT with different prompt designs: 1) SGEAT (standard) uses no prompt and performs unconditional generation.
2) SGEAT (heuristic) uses a set of manually crafted prompts inspired by the definition of toxicity from Perspective API.
We discuss the set of considered templates in Appendix §B and report the one that achieves the lowest toxicity in our experiments. 3)
SGEAT (augmented) constructs prompts that tend to yield nontoxic continuations.
Specifically, we find the most nontoxic documents from the unconditional generation, and split each document into half as the prompts and the continuations.
In this way, we obtain the prompts that are highly likely to generate nontoxic language.
SGEAT (augmented) can also be regarded as a data augmentation of
SGEAT (standard) from the nontoxic distribution.
We present more details in Appendix §B.
Self-Generation uses the prompts from the last step to generate up to 1,000 tokens and truncate all the sentences at the end-of-document (EOD) token once generated.
We use nucleus sampling [36] with p = 0 .
9 and the temperature of 1 during generation.
To demonstrate the data efficiency of
SGEAT, we generate only 100k documents in total, in comparison with DAPT in Gehman et al.
[10] that uses 7500k documents from the pre-training corpus.
Data Filtering further filters out toxic samples to ensure the training corpus is mostly nontoxic.
Specifically, we follow the standard DAPT setup in Gehman et al.
[10] and use Perspective API to annotate the toxicity of the raw generated text.
Different from DAPT that performs aggressive filtering on pre-training data and only keeps the most nontoxic 2% of the documents, we keep the most nontoxic 50% of the generated text to demonstrate the quality and data efficiency of SGEAT.
We present the curated data toxicity and statistics in Appendix Table 13.
Domain-Adaptive Training leverages the curated nontoxic corpus to further fine-tune the pre-trained
LM with standard log-likelihood loss and adapt it to the nontoxic data domain.
We present more training details in Appendix §A.2.
4.2 Evaluation Results of Domain-Adaptive Training
In this subsection, we evaluate existing domain-adaptive training methods on 1.3B LM (similar to
GPT3-XL), and discuss the impacts of model sizes in Section 5.
Baselines: We consider the following domain-adaptive training baselines: DAPT (nontoxic) [30] uses a nontoxic subset of pre-training corpus annotated by Perspective API to perform domainadaptive training; and Jigsaw (nontoxic) uses a human-annotated nontoxic subset of Jigsaw Toxic
Comment Classification dataset 6 .
We present the evaluation results in Table 2.
Among all domain-adaptive training methods, we find that SGEAT (augmented) achieves the lowest toxicity scores with moderate perplexity increases and without degrading the LM utility accuracy (or even improving).
Specifically, SGEAT (augmented) reduces the toxicity of the standard 1.3B by 0.14 at the cost of a slight PPL increase and does not hurt the utility of LMs on downstream tasks.
Moreover, we note that although DAPT (nontoxic) usestimes larger corpus than SGEAT (augmented) (shown in Appendix Table 13), SGEAT (augmented) still achieves lower toxicity than DAPT (nontoxic), which implies that self-generated data has better data efficiency for domain-adaptive training.
We think such high data efficiency comes from the fact that i ) the self-generated corpus well captures the high-density regions of the output space of a pre-trained LM, and ii ) training on autoregressively generated corpus mitigates the exposure bias [20, 21], which refers to the train-test discrepancy of an autoregressive model.
Thus, when we train the LM on the self-generated non-toxic corpus, it tends to increase the likelihood on the non-toxic density region, which enables data-efficient training to detoxify the model.
The human-annotated nontoxic Jigsaw dataset fails to detoxify the LM and even increases the model toxicity.
We speculate the major reason is that the nontoxic subset of the Jigsaw dataset has a much higher average data toxicity than SGEAT, as shown in Appendix Table 13.
Among SGEAT methods, we observe that SGEAT (augmented) achieves the best detoxification result at a similar level of PPL increase, while SGEAT (heuristic) is less effective to detoxify the LM.
We think the reason lies in the data diversity: The unconditional generation covers the diverse regions of the generation distribution and yields the most diverse data distribution, and thus SGEAT (standard) also achieves good detoxification performance.
In contrast, SGEAT (heuristic) uses only a single prompt for generation, which limits the diversity of the generation.
More analysis about prompt design is in Appendix §B.6.
4.3 Evaluation Results of Decoding-time Methods
Besides the domain-adaptive training baselines, we also compare with decoding-time algorithms:
Word Banning [10] sets the probability of generating any word from a list 7 of profanity, slurs, and swearwords to zero during decoding.
Rejection sampling [15, 13] generates up to K samples given each prompt until we obtain a nontoxic sample, otherwise we return the sample with the lowest toxicity score from Perspective API.
We set K = 4 due to the computational limit.
DE XPERTS [14] is the state-of-the-art decoding-time algorithm for detoxification that uses two auxiliary expert and anti-expert LMs to steer a model’s generation.
The expert model is the same as DAPT (nontoxic); while the anti-expert model is fine-tuned on the top toxic portion of OWTC with 150k documents.
When comparing domain-adaptive training methods with decoding-time methods.
We note that rejection sampling adds 4 × computational overhead during decoding, but is less effective than domain-adaptive training SGEAT, as LM rarely generates nontoxic continuations given toxic prompts
[Table]
Although the state-of-the-art DE XPERTS achieves significantly lower toxicity scores than
SGEAT, we also observe that there is a concerning perplexity and utility degradation, with an increase of 9.47 in PPL and a drop of 9 . 4% in downstream task accuracy.
Such degradation makes the detoxified 1.3B LM quality even worse than a standard 126M LM, as shown in Table 1.
We hope that our findings can motivate researchers to focus more on the trade-off between detoxification and LM quality when designing detoxification algorithms.
Since decoding-time algorithms are orthogonal to domain-adaptive training methods, it is easy to combine both methods together.
Specifically, we replace the standard 1.3B model used in rejection sampling and DE XPERTS with SGEAT (augmented) detoxified one, and observe that the combined method can yield the lowest toxicity scores among existing methods.
Impact of Model Size
We next investigate how the number of model parameters impacts the domain-adaptive training for detoxification.
Specifically, we show that 1) models with different number of parameters trained on the same pre-training corpus display similar levels of toxicity; 2) self-generated data consistently demonstrates better detoxification effectiveness than pre-training corpus across different parameter sizes; 3) larger LMs require more efforts to reduce the toxicity.
Standard Model Toxicity.
We first evaluate the toxicity of 5 standard LMs across different parameter sizes in Table 1 and Table 9.
We observe that the standard LMs, pre-trained on the same pre-training data with different parameter sizes, display similar levels of toxicity.
It suggests that the toxicity comes from the dataset, instead of the model size .
Detoxification Effectiveness of SGEAT.
We then evaluate our best SGEAT (augmented) and compare with the best domain-adaptive training baseline DAPT (nontoxic) in Table 3.
We note that SGEAT consistently outperforms DAPT over different sizes even when using 1/3 smaller training corpus.
For example, SGEAT (augmented) can reduce the toxicity probability from 57% to 30% for the 126M
LM, 7% lower than DAPT.
These results confirm that: the self-generated corpus is more efficient to detoxify the LM than using the curated corpus of pre-training data.
Larger-scale LMs requires more endeavors to detoxify.
From Table 3, we observe the detoxification effectiveness decays for both DAPT and SGEAT with the increase of LM parameter sizes.
For instance, the toxicity probability of the 530B SGEAT LM is only the 16% lower than the standard 530B LM, compared to the drop of 27% toxicity probability for the 126M one.
We figure the potential reason of such small improvement on larger LM is that large LM tends to require more training data and fine-tuning epochs to detoxify.
Therefore, we conduct additional experiments on the 530B LM, by either increasing the training epochs from 3 to 5 or generate more data from 50k to 100k samples for adaptive training.
We find that while both methods further reduce the toxicity of the 530B LM, training for more epochs might lead to model overfitting and hurts the PPL and downstream accuracy by a large margin.
In contrast, training with more data demonstrates a better trade-off between detoxification and LM quality.
It implies that it needs more endeavors to detoxify large-scale LMs.
[Table] [Table]
[Figure]
LM Quality Evaluation.
We also evaluate whether domain-adaptive training impacts the perplexity and utility of LMs in Table 3.
When trained within 3 epochs, we find that the PPL of LMs slightly increases and the LM utility drops a little in most cases, which suggest that models gradually adapt to the nontoxic domain without a significant sign of overfitting or degradation in terms of LM quality.
[Figure]
Domain Adaptation v.s. Overfitting.
We visualize the trade-off at different training phases in Figure 2 for 530B LM.
Specifically, we record the validation perplexity and model toxicity after 1, 3, and 5 training epochs for DAPT (nontoxic, 150k) and SGEAT (augmented, 50k) .
We also add a curve DAPT (nontoxic, 50k) , which samples 50k documents from DAPT (nontoxic, 150k) to have a fair comparison with SGEAT (augmented, 50k).
We observe that at the beginning of training, the model toxicity drops substantially and barely sacrifices the model PPL (steep slope).
Then it is gradually adapted towards the nontoxic domain.
SGEAT demonstrates a better trade-off between toxicity and quality, as SGEAT achieves substantially lower toxicity with the same PPL afterepoch of training.
Finally, we observe the curve is becoming more flat, especially for DAPT, which indicates the transition from the domain adaptation to overfitting.
For LMs with different sizes fine-tuned with different methods, we find 3 epochs is a good cut-off point for whole model adaption, which achieves good trade-off between model toxicity and perplexity.
This rule of thumb is also aligned with previous study [10].
Parameter-efficient Training
To cope with the challenges of large-scale LMs, we explore two parameter-efficient training paradigms: adapter [18] and prefix tuning [19], and evaluate whether they can improve the LM quality and achieve a better trade-off between detoxification and LM quality than whole model adaption.
We show that: in the scenario of detoxification, 1) adapter demonstrates a better trade-off than prefix tuning, and 2) adapter can further mitigate the drop of LM quality and improve the trade-off upon whole-model adaptation for large-scale LMs.
6.1 Comparison between Adapter and Prefix Tuning
Both adapter and prefix tuning add additional parameters to the standard LM, and only optimize the added parameters during training without perturbing the original LM parameters.
Such paradigm provides the flexibility, especially for large-scale LMs, to adapt to different domains with a few additional parameters, rather than heavily fine-tune the whole model with multiple copies of the whole model parameters for different domains.
In this study, we further investigate whether such training schemes can provide more advantages to detoxify LMs.
Adapter [18] adds additional bottleneck projection layers to each transformer layer with residual connections.
At the beginning of the training, the projection layer is initialized to almost zero to improve the training stability.
Prefix tuning [19] appends additional continuous “prefix” vectors to the input to better steer LMs’ generations.
To have a comprehensive understanding and comparison between adapter and prefix tuning, we first perform ablation studies on small-scale 1.3B LM over the key hyper-parameters: the projection size for adapter and the prefix length for prefix tuning.
We follow the same training schedules as whole model adaptation but train more epochs so that the PPL reaches a similar level as whole model adaptation.
We present the evaluation results in Table 4.
When comparing Table 4a with Table 4b, we observe that adapter demonstrates a better trade-off between detoxification and LM quality than prefix tuning.
We figure the possible reasons are two folds: 1) given the same projection size and prefix length, the number of additional parameters of adapter is around twice more than prefix tuning, which gives more capacity for adapter to perform domain adaptation; 2) however, while longer prefix length could give more capacity to steer the model generation, it also adds too many irrelevant contexts, which not only hurts the perplexity of the
LM but also slows down the decoding speed.
Compared to the whole model adaption, adapter does not show significant advantages in terms of detoxification and LM quality for small-scale models like 1.3B one.
For adapter results with different projection sizes, we observe that a larger projection size yields better detoxification effectiveness possibly due to larger model capacity.
We thus apply adapter with the projection size = 1024 to larger-scale LMs (8.3B and 530B) and investigate whether it can solve the challenges of large-scale LMs.
6.2 Apply Adapter to larger-scale Models
We follow the same training schedules as the whole model adaptation to train the adapters for largerscale LMs.
We stop training when they reach similar levels of toxicity as the whole model adaptation, and evaluate the perplexity and utility of LMs in Table 5.
We can see that for larger-scale LMs, adapter can not only improve the parameter efficiency, but also mitigate the PPL and the LM quality drop.
In particular, for the 530B model, adapter can mitigate the drop of PPL for at most 0.64 and improve the average downstream task accuracy by 0 .
Human Evaluation
We further verify our findings via human evaluation on the standard models, DAPT, SGEAT, and decoding-time algorithm DE XPERTS across five LM sizes.
[Figure]
Setup.
We sample the 300 prompts from R EAL T OXICITY P ROMPT benchmark while keeping the ratio of toxic and nontoxic prompts to 1:3 as the same as the full set, and evaluate the continuations of each model.
We follow
Welbl et al.
[15] to ask LMs to generate up to 100 tokens and avoid incomplete sentences and collect the most toxic continuations via
Perspective API over 25 generations.
Finally, we gather 5,700 continuations from 19 models and randomly shuffle them for human evaluation.
Then we group samples into a batch of 10, and assign them to 5 annotators.
In total 187 workers from Amazon MTurk participated in the evaluation.
To consider the annotators’ well-being, we make sure the average number of toxic samples (T OXICITY >= 0.5 evaluated by Perspective API) is less than or equal to 3 in each batch of 10 samples.
To calculate the average scores of annotations, we follow Welbl et al.
[15] to map “Very Toxicity” and “Toxic” to 1 , “Not Toxic” to 0 , and discard “Not Sure“ annotations.
We average the scores from 5 annotators for each sample and then report the averaged number over the 300 prompts in Figure 3.
The detailed scores can be found in Table 8 in Appendix.
We present more details in Appendix §A.3.
By comparing the objective evaluation with human evaluation,
[Table]
we observe that the toxicity scores from the human evaluation are mostly aligned with objective evaluation via Perspective API.
Such findings are also confirmed by Welbl et al.
The human evaluation also verifies that i ) LMs of different sizes have similar levels of toxicity, and ii ) LMs of larger sizes present more challenges to detoxify.
Discussion 8.1 Bias against Marginalized Groups
We follow the setting of Welbl et al.
[15] and evaluate the PPL of the 1.3B standard LM and
SGEAT (augmented) fine-tuned LM on the gender and ethnicity domains using the BOLD dataset [37] as shown in Table 6.
The former contains Wikipedia sentences about female and male actors, and the latter domain contains sentences about people with different ethnic backgrounds [15].
We find that: ( i ) LM PPL increases moderately on the BOLD dataset after effective detoxification, which is aligned with our findings in §4.2.
( ii )
There is no noticeable discrepancy of PPL increase among male and female in the gender domain, which suggests that SGEAT does not exacerbate the gender biases.
( iii )
There is a higher PPL increase for the Hispanic group than other demographic groups in the ethnicity domain.
We hypothesize that such bias mainly comes from the pre-training model and corpus, because the pre-trained Standard model already has much higher perplexity for Hispanic group.
Our findings partly align with recent findings on the trade-off between detoxification and bias [13, 15].
We leave it as an important future direction to mitigate the social biases of pre-trained foundation models, as well as design new approache that jointly reduce toxicity and racial bias.
8.2 Limitation of SGEAT
While we observe that SGEAT has demonstrated very good trade-off between detoxification effectiveness and perplexity, SGEAT still has potentials to further improve.
Bias within Hate Speech Detector.
Similar to DAPT, SGEAT also relies on a hate speech classifier ( i.e., Perspective API) to filter out toxic samples.
However, existing classifier on toxicity classification is imperfect and is known to amplify the social bias against different demographic groups due to the annotation bias and sampling bias [13] ( e.g., the classifier tend to assign higher toxicity scores for text mentioning historically underrepresented groups).
As a result, SGEAT may also be impacted due to the use of Perspective API, which may filter both toxic text and minority identity mentions.
Nevertheless, we believe that SGEAT can get more benefits with a more robust, unbiased, and fair hate speech detector, so models fine-tuned on the filtered corpus can unlearn toxicity without forgetting corpus from minority groups.
Bias within Pre-trained Model.
As discussed in § 8.1, we observe pre-trained models already exhibit bias against certain demographic groups.
As a result, the self-generated corpus may inherit the bias and harm the coverage of detoxification.
Thus we leave it as an important future direction to build a bias-free pre-trained LM, which can benefit SGEAT and other detoxification methods.
Conclusion
We explore the limits of domain-adaptive training for detoxifying LMs along three aspects: 1) training corpus; 2) model size and 3) parameter-efficient training.
We first identify the trade-off between detoxification effectiveness and LM quality in detoxification methods.
We propose Self-Generation
Enabled domain-Adaptive Training (SGEAT), which leverages the generative power of LMs for data-efficient and effective detoxification.
We comprehensively detoxify LMs with parameters sizes ranging from 126M up to 530B and find interesting properties of large-scale LMs.
We demonstrate that adapter provides parameter-efficient training and achieves a better trade-off of toxicity and LM quality.
We hope our work can shed light on the development of detoxification techniques that can largely reduce toxicity while maintaining good perplexity and downstream task accuracies.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 8.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Our work focus reducing the toxicity of LMs.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix Table 11 and(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] See Appendix A. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[Yes] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] See Appendix A.
Appendix A Experimental Details A.1 Details of Pre-trained LMs
The architecture details of pre-trained LMs are in Table 7.
The corresponding perplexity and downstream task accuracy is shown in Table 9.
[Table]
A.2 Training Details
Throughout our experiments, we follow the setup of Gehman et al.
[10] and fine-tune the LMs forepochs.
We use the batch size equal to 512 with sequence length 2048.
We use the Adam optimizer (epsilon=1e-5, beta-1=0.9, beta-2=0.95) with initial lr = 1e-5 for 530B LM, while we use lr = 2e-5 for all other LMs.
We set weight decay to 0.1 for all LMs.
Our experiments are conducted on the DGX
A100 servers with 8x A100 GPUs.
A.3 Details of Evaluation Protocols
Details of Human evaluation Asking humans to annotate toxicity necessarily exposes them to toxic language.
Before conducting our study, it was reviewed by our Institutional Review Board (IRB).
Participants were recruited through Amazon MTurk.
Since toxicity evaluation is subjective to different social groups, we do not have any constraints on the demographic background of annotators.
Since our toxicity evaluation mainly focuses on English, we also constrain the regions of annotators to the United States, Canada, Australia, and the United Kingdom.
To improve the quality of the annotations, we require the participated annotators to have at least 1000 approved HITs and a lifelong
HIT approval rate is greater than 98% .
We group continuations in a batch of 10 samples and assign them to annotators.
The payment rate for each batch of samples is $0 .
We follow the definition of Perspective API and Welbl et al.
[15] on toxicity as follows:
• Very Toxic: Text that is very hateful, aggressive, disrespectful, or otherwise very likely to make a user leave a discussion or give up on sharing their perspective.
• Toxic: Text that is rude, disrespectful, unreasonable, or otherwise likely to make a user leave a discussion or give up on sharing their perspective.
• Not Toxic: Neutral, civil, or even nice text very unlikely to discourage the conversation.
We adapt the instructions from Welbl et al.
[15] and show the annotation instructions on Amazon
MTurk platform in Figure 4 and Figure 5.
We put the detailed human-annotated toxicity scores in Table 8 and compare with Perspective API.
We observe that the toxicity scores from the human evaluation are mostly aligned with objective evaluation via Perspective API.
Details of PPL Evaluation We evaluate the LM PPL on a held-out validation set from the pretraining corpus.
Note that, the validation set can be different from the one in Smith et al.
[6] due to different random seed and chunking.
Details of Downstream Task Evaluation We consider the following 9 downstream tasks:
• ANLI [38] is a large-scale NLI adversarial benchmark dataset.
• BoolQ [39] is a question answering dataset for yes/no questions.
• Hellaswag [40] is a commonsense NLI dataset.
• LAMBADA [41] is a cloze test (word prediction) dataset.
• PIQA [42] is a physical commonsense reasoning and a corresponding benchmark dataset.
• RACE [43] is a large-scale reading comprehension dataset.
• WiC [44] is a multilingual Word-in-Context Dataset for the evaluation of context-sensitive word embeddings.
• WinoGrande [45] is commonsense reasoning for pronoun resolution problems.
Our evaluation code is based on Gao et al.
[Table] [Table]
B Details of Prompt Design
Our prompt exploration starts from unconditional generation
[Table]
and then moves on to the conditional generation scenarios for investigating which prompts can best facilitate LMs to generate high-quality nontoxic data.
B.1 Unconditional Generation
By only taking the start-of-sentence token 8 as input
[Table]
unconditional generation unveils the intrinsic self-generation quality given no prompt and thus tends to yield the most diverse data distribution.
During the generation
[Table]
we use different random seeds to collect a total of 100,000 self-generated samples as SGEAT (standard).
B.2 Conditional Generation
Prompts provide language models with context to generate a variety of desired contents.
For detoxification purposes, we aim to design prompts such that language models can generate highquality and diverse nontoxic data.
Manually-crafted Prompts (SGEAT (heuristic))
We start with the definition of toxicity following
Perspective API as the seed template.
Then we manually craft prompts to perform self-generation and evaluate their detoxification performances.
Specifically, we follow the definition of toxicity from Perspective API as well as the inspiration by the recent work from Prompt Engineering [46, 47, 19, 48–51] that repeating the prompts and prompting
LMs in the format of Question Answering, and design the prompts below to study the generation and understanding power of the LMs,
Negative Prompt (for once).
The following text does not contain rude, disrespectful, and unreasonable languages: ___. 2.
Positive Prompt (for once).
The following text contains kind, respectful, and reasonable languages: ___. 3.
Positive Prompt twice.
Repeating the above prompt twice.
Positive Prompt in the format of QA.
Question :
The following text contains kind, respectful, and reasonable languages.
Answer : ___. 5.
Positive Prompt twice in the format of QA.
Repeating the above prompt twice.
For each prompt, we generate 100,000 continuations, and use them as a self-generated dataset before data filtering.
Automatically-constructed Prompts (SGEAT (augmented))
As we will show in the experiment, manually-crafted prompts suffer from the limited templates and may impact the diversity of the self-generated data.
Inspired by Gehman et al.
[10], we design the following process to automatically construct prompts: i)
We use the unconditional generated data in §B.1 as the source data to construct prompts, as it represents the distribution of LM while keeping the data diversity.
ii)
We apply
Perspective API to annotate the unconditionally generated data, and select the most nontoxic 25% data, which in total collects the most nontoxic 25,000 documents.
iii)
We split each document into two parts, yielding 25,000 prompts and continuations.
For each prompt, we generate 4 continuations and eventually collect 100,000 self-generated samples before data filtering.
We list a glossary of prompts (automatically-constructed prompts and manually-crafted prompts) we considered along with their explanation and examples in Table 10.
B.3 Unprompted Toxicity
During unconditional generation to construct prompts, we can also evaluate the unprompted toxicity (i.e., unconditional generation scenario) of LMs of different sizes.
We present the unconditional generated data toxicity in Table 12.
We can see that standard LMs yield similar levels of data toxicity in the unconditional generation scenario, which is also aligned with the observations in our main experiment.
B.4 Data Toxicity given Automatically-constructed Prompts
Furthermore, given our automatically-constructed prompts, our SGEAT demonstrates lower data toxicity than an unconditional generation, as shown in Table 12.
B.5 Data Diversity Evaluation
Data diversity is also an important factor that can impact the detoxification effectiveness.
To avoid generating duplicated data, we use nucleus sampling [36] with p = 0 .
9 with different random seeds, which significantly reduces the probability to generate duplicated output.
Specifically,
[Table]
Examples
[Table]
The following text does not contain rude, disrespectful, and unreasonable languages: ___________________________________.
The following text does not contain rude, disrespectful, and unreasonable languages: Black Mountain College in North Carolina awarded the prize last year to a team that used algorithms to predict whether a ball would turn into a goal.
A world record this time will be considered statistically significant.
The team, from the
Netherlands, started by setting out the locations of six ball goals in three different games: grass (A); turf (B); and cobblestone (C)...
( Explanation : The prompt is expressed using negation “does not”.)
[Table]
The following text contains kind, respectful, and reasonable languages: ___________________________________.
The following text contains kind, respectful, and reasonable languages: (0 / 1) Review score is calculated from customers who have booked through our site.
Add Your Review This review is the subjective opinion of a TripAdvisor member and not of TripAdvisor LLC.
The hotel itself is a fairly large and nicely maintained property, the rooms are a little dated but comfortable, with clean and useful toiletries (there’s a decent sized vanity), well-appointed bathrooms, and each room has a fridge and tea/coffee maker..
( Explanation : The prompt is expressed via the positive tone.)
[Table]
The following text contains kind, respectful, and reasonable languages: The following text contains kind, respectful, and reasonable languages: ___________________________________.
The following text contains kind, respectful, and reasonable languages: The following text contains kind, respectful, and reasonable languages:
These articles discuss the following topics: 1.
the 44th Annual Meeting of the American Association of Applied Science.
October 28-30, 2012 in the Omni Shoreham
[Table] [Table]
[Table]
this setting will have on average more than 200 candidate tokens to sample at each step, and we generate up to 1000 steps.
Thus the likelihood of generating duplicated data should be very small.
To further verify the findings, we evaluate the diversity of SGEAT (heuristic), SGEAT (standard), and
OWTC using distinct-1, distinct-2, distinct3, and distinct-4, which measures the number of distinct n-grams of the corpus [52].
The results are shown in the Table 14.
We find that SGEAT (heuristic) indeed generates less diverse data than SGEAT (standard), and thus limits the effectiveness of detoxification.
In contrast, the diversity of SGEAT (standard) is relatively close to the real-world corpus OWTC.
B.6 Benchmark and Analysis of Prompt Design
As the core of SGEAT is the prompt design, we perform a systematic study on the 1.3B LM to evaluate how different prompts impact the self-generated data quality, which further affects the detoxification performance.
We evaluate the prompts following two fronts: i ) Data Toxicity , which directly evaluates the generated data toxicity scores via Perspective API in Table 13.
Specifically, we report the average toxicity of the generated data, the probability of generating toxic and nontoxic samples, their corresponding toxicity, and their toxicity scores after filtering; and ii ) Model Toxicity , which evaluates the final performance fine-tuned with the generated data in Table 11.
Analyzing both Table 11 and 13, we have the following observations: i)
Using all automaticallyconstructed prompts provides the best toxicity reduction performance among all the prompt designs.
[Table] [Table]
This result is also aligned with the observation in Table 13 that automatically-constructed prompts yield the least average data toxicity (0.07).
ii) Low data toxicity does not necessarily lead to good model toxicity after fine-tuning.
Diversity also matters.
When we choose the least nontoxic prompt from automatically-constructed prompts as the single prompt for generation, we find that although the generated dataset achieves the average data toxicity as low as 4e-4, the toxicity reduction is not as effective as using all automatic-constructed prompts.
We think the reason is that both data toxicity and data diversity contribute to the detoxification effectiveness.
The prompts with lower data toxicity can more effectively pull the generation distribution from the toxic domain to the nontoxic domain, while the higher prompt diversity can cover more regions of the generation distribution, thus yielding lower model toxicity.
iii) Manually-crafted prompts are not enough to generate high-quality non-toxic data.
Therefore
, manually-crafted prompts yield worse detoxification effectiveness than unconditional generation.
The unconditional generation covers the diverse regions of the generation distribution and yields the most diverse data distribution, and thus also achieves good detoxification performance.
In contrast, human-crafted prompts use only a single prompt for generation, which limits the diversity of the generation.
Moreover, the generation tends to follow the topics of the prompts related to toxicity, and thus is more likely to yield toxic samples than unconditional generation, as shown in Table 13.
We also note that repeating the positive prompt twice can cause
lower toxicity in the continuations, while prompting the language model in the question-answering format [53] is less helpful for generating lower toxicity data.
In addition, using negative prompts may even backfire and increase the model toxicity, suggesting that it is better to prompt language models in a positive way instead of using negations.
iv) Human-annotated nontoxic Jigsaw dataset fails to detoxify the LM, and even increases the model toxicity.
We think there are two main reasons: 1) the nontoxic subset of the Jigsaw dataset has much higher data toxicity than the filtered OWTC; 2) the Jigsaw data has some domain shift from the pre-training data distribution, and thus limits the effectiveness for detoxification
C Additional Experimental Results C.1 Downstream Task Accuracy
We present the detailed downstream task accuracy of each method for nine tasks in Table 15, 17, 16, and 18.
[Table]
[Table]
C.2 Perplexity Evaluation on Nontoxic Validation Set
Hypothesis We hypothesize that the reasons for the perplexity increase on the validation set of the pre-training data after domain-adaptive training are two fold: 1)
The validation set may contain toxic language, while the LMs are already adapted to the nontoxic domain.
Thus it is expected that the
LM loss on the toxic portion increase after detoxification, which leads to the PPL increase on the full validation set.
The filtered non-toxic corpus are not perfect (e
[Table]
g., poor coverage of language for different topics), which may hurt the LM’s quality after domain-adaptive training.
This is also confirmed by the degradation of down-stream task accuracy.
To verify the hypothesis
[Table]
we further filter our validation set based on Perspective API to construct several nontoxic corpora
[Table]
and evaluate the LM PPL on these nontoxic corpus.
Setup We construct three validation set with different filter rates as shown in Table 19, where
Nontoxic @
x% refers that we keep the most x% of nontoxic documents for PPL evaluation.
We also present the PPL evaluation on Nontoxic @ 10% for all detoxification methods we consider for the 1.3B model in Table 20.
Analysis We find that: 1)
The PPL increase on the nontoxic subsets of validation corpus is less than that on the full validation set.
This suggests that the toxic documents in the validation set indeed lead to some of the PPL increase for our detoxified language models.
The lower the average toxicity score the validation set has, the less PPL increases.
The trend of PPL increase on nontoxic corpus is almost the same as that on the full validation set.
Thus we report the standard PPL increase on our full held-out set in our main paper to reflect the level of LM quality degradation.
C.3 Perplexity Evaluation on Self-Generated Data v.s. Pre-training Data
Hypothesis We think such high data efficiency comes from the fact that i ) the self-generated corpus well captures the high-density regions of the output space of a pre-trained LM, and ii ) training on
[Table]
[Table] [Table]
[Table]
autoregressively generated corpus mitigates the exposure bias [20, 21], which refers to the train-test discrepancy of an autoregressive model.
Setup We leverage the PPL to verify it.
If the generated corpus shows a lower PPL, it means that the corpus better captures the high-density region of the LM.
We evaluate and compare the PPL of the generated corpus of SGEAT (augmented) and OWTC by the standard 1.3B LM.
Analysis We find that SGEAT (augmented) demonstrates a much lower PPL (5.98) than OWTC (7.93), which confirms our hypothesis that our generated corpus SGEAT (augmented) better captures the high-density regions of the LM output space.
C.4 Transferring Self-Generated Dataset from Larger Models to Smaller Models
We fine-tune a 126M model with the 1.3B generated corpus SGEAT (augmented) following the same training strategy.
We evaluate the expected maximum toxicity and perplexity and compare with the 126M fine-tuned with 126M generated corpus SGEAT (augmented).
The results are shown in Table 21 below.
[Table]
In terms of toxicity reduction, we observe that using the generated corpus from a larger LM to fine-tune a smaller LM is not as effective as using the self-generated corpus, which emphasizes the importance of fine-tuning with self-generated data to mitigate the exposure bias.
However, the corpus generated from 1.3B LM does have better language quality than 126M and is closer to the pre-training corpus, thus leading to a better validation PPL than the self-generated corpus.
C.5 Mixing Nontoxic Pre-training Corpus and Self-Generated Data
We fine-tune the mixed dataset of DAPT and SGEAT (augmented) with the mixture ratio of 0.5 (half of the documents are sampled from DAPT, and the other half are sampled from SGEAT (augmented)).
We follow the same training schedules and iterations of DAPT (nontoxic) to fine-tune the LM on the mixed dataset.
The results are shown in the Table 22 below.
[Table]
From the results, we can see that the detoxification effect of the mixed SGEAT and DAPT is an interpolation of both methods.
We agree that combining these two methods does yield a better trade-off, and provides a flexible way to control the PPL and toxicity by controlling different mixture ratios.
D Discussion and Case Studies D.1 Guidance on which adaptation methods to choose
As shown in Appendix C.5 as well as the main experiments, we summarize our thoughts on how to select the adaptation methods here: we suggest that using SGEAT with adapters can be an efficient method to achieve a good trade-off between toxicity and PPL; 2) however if valid PPL is very important in some applications, we suggest that mixing DAPT with SGEAT and performing parameter-efficient training can achieve better PPL and satisfactory toxicity as well.
D.2 Error Analysis of Perspective API
Although Perspective API and human annotation are statistically highly aligned in most cases as confirmed in our §7 and recent work[15], we also note that in some cases, Perspective API can also make mistakes.
For example, Perspective API can overestimate the toxicity of test samples.
[15] find that the nontoxic continuations from detoxified models verified by human annotators are sometimes classified as highly toxic by Perspective API.
We also confirm the same phenomena in our experiments.
Quantitatively, we find that among all toxic (classified by Perspective API) samples generated by
DE XPERTS + SGEAT, around 34% samples are actually annotated as nontoxic by the majority vote of 5 human judges.
Qualitatively, we show a qualitative example of failure cases of Perspective API below: “Let us help you with your cleaning!
Spend your free time doing what you want to do, not what you feel you need to do.
Established in 1987, SelClene Herts pride themselves on supplying reliable domestic cleaners that you can trust.
The reputation of SelClene (St Albans) Ltd has been built up over the years by our many happy customers across Hertfordshire and Buckinghamshire.
We have a good supply of domestic cleaners across the area to do your housework and keep your home spick and span.
Our customers across Herts & Bucks use our cleaning services because of our expertise and customer service.”
The above paragraph comes from the pre-training corpus and it is surprisingly the most “toxic” document classified by Perspective API with toxicity score equal to 0 .
While we have submitted the error case to Perspective API, we hope our findings can motivate the development of more robust and unbiased hate speech detection model.
D.3 Case Studies of SGEAT
Warning!!
The following content contains toxic language.
Now, we present both successful and failure cases of detoxification using SGEAT (augmented) for 530B LM in Table 23 and Table 24.
[Table] [Table]