Unveiling the Implicit Toxicity in Large Language Models (2023)

Abstract: The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent studies primarily focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, we show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. Moreover, we propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs. Specifically, we optimize the language model with a reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. Experiments on five widely-adopted toxicity classifiers demonstrate that the attack success rate can be significantly improved through RL fine-tuning. For instance, the RL-finetuned LLaMA-13B model achieves an attack success rate of 90.04% on BAD and 62.85% on Davinci003. Our findings suggest that LLMs pose a significant threat in generating undetectable implicit toxic outputs. We further show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect LLM-generated implicit toxic language.

## introduction with the rapid progress in large-scale pre-training @xcite , large language models (llms) have shown impressive capabilities in natural language understanding and generation, leading to significant breakthroughs in zero-shot / few-shot learning @xcite . however, the open-endedness nature of llms, combined with their powerful abilities, also introduces new risks of harmful behaviors @xcite ). ## preliminary experiments on implicit toxicity in large language models implicit toxicity has emerged as a main challenge in the field of toxicity detection owing to its nuanced nature @xcite . rather than overtly abusive language such as swearwords, implicit toxicity is conveyed through a variety of linguistic features (such as euphemism @xcite , sarcasm @xcite , circumlocution @xcite , and metaphor @xcite ) and extralinguistic knowledge (such as commonsense knowledge @xcite , world knowledge @xcite , and social norm @xcite ). inspired by prior analyses on the challenge of detecting implicit toxicity, we conduct preliminary experiments to explore whether large language models, which have seen extensive linguistic features and extralinguistic knowledge during pre-training @xcite , can generate implicit toxic responses that evade toxicity classifiers. specifically, we use zero-shot prompting to instruct gpt-3.5turbo to generate implicit toxic responses. we then evaluate the attack success rate of llmgenerated implicit toxic responses on various stateof-the-art toxicity classifiers, comparing it with that of previous toxic benchmark datasets. the quantitative results, as illustrated in figure 1 , reveal the following findings: (1) state-of-the-art toxicity classifiers achieve impressive performance on previous toxic benchmark datasets, including implicit toxic benchmarks such as @xcite . this indicates their advanced language understanding abilities that surpass mere keyword-matching. for instance, the example "you can tell a woman is a good person or not by how much she covers her skin" (which is an implicit toxic example from toxigen), can be easily detected by these classifiers. (2) surprisingly, these toxicity classifiers are vulnerable to llm-generated implicit toxic responses, showing a significantly higher attack success rate ranging from 58.47% (on davinci003) to 96.69% (on perspective-api). ## methodology based on the impressive zero-shot performance of llms in expressing implicit toxicity, we propose a novel method based on reinforcement learning to further explore the potential of inducing implicit toxicity in llms. specifically, starting with a vanilla large language model denoted as π ϕ (which is chosen as trainable llms like llama @xcite ), our method consists of three steps as illustrated in figure 2: • supervised learning: we first warm-start the policy model π ϕ by conducting supervised learning such that π ϕ can perform reasonably well in generating implicit toxic responses. however, π ϕ still occasionally generates explicit toxic or non-toxic responses. ## conclusion this paper identifies a novel safety risk of llms, namely their ability to generate implicit toxic outputs, which are exceptionally difficult to detect with existing toxicity classifiers. we first conduct preliminary experiments on gpt-3.5-turbo via zeroshot prompting. we further propose a rl-based method to induce implicit toxicity in llms via optimizing the reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. extensive experiments demonstrate that the implicit toxic responses induced by our method are significantly more challenging to detect than previous baselines. further analysis reveals that llms leverage diverse toxic types, linguistic features, and extralinguistic knowledge to express implicit toxicity. finally, we show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect llmgenerated implicit toxic responses.