Toxicity in chatgpt: Analyzing persona-assigned language models (2023)

Abstract: Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Legislation has recognized its significance and recently drafted a “Blueprint For An AI Bill Of Rights” which calls for domain experts to identify risks and potential impact of AI systems. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6\times , with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others ( 3\times more) irrespective of the assigned persona, reflecting discriminatory biases in the model. Our findings show that multiple provisions in the legislative blueprint are being violated, and we hope that the broader AI community rethinks the efficacy of current safety guardrails and develops better techniques that lead to robust, safe, and trustworthy AI.

## introduction large language models (llms) like @xcite and palm @xcite have shown impressive potential on a multitude of complex tasks like writing essays and poems, engaging in dialogue, and generating code. these abilities coupled with the availability of apis have accelerated the adoption of llms in numerous toxicity score figure 1: (top) an example of persona-assigned chat-gpt that is assigned the persona of the boxer muhammad ali. (bottom) chatgpt not only generates toxic language but also exhibits variation in the degree of toxicity depending on the persona. for example, significantly more toxic language can be generated using chatgpt by setting its system parameter, or in other words persona, to that of adolf hitler. we find that chatgpt violates both the aforementioned provisions detailed in the blueprint. contrary to the results of prior studies @xcite , we find that chatgpt can be consistently toxic about a wide range of topics when it is assigned a persona. chatgpt can be assigned a persona by setting its system parameter, a provision of the chatgpt api that influences the nature of chatgpt throughout the conversation. see fig. 1 (top) for an example of setting the system-level parameter -here, when chatgpt's persona is set to that of the boxer "muhammad ali", its toxicity increases ∼3-fold when compared to chatgpt with default system settings. this is particularly worrying as technologies that build on top of chat-gpt can generate toxic language by making such system-level modifications. in order to systematically analyze and understand this behavior of chatgpt, we perform an extensive study of the toxicity in its generations, especially when assigned different personas through the system parameter. we consider an extensive list of 90 personas assigned to chatgpt and analyze its responses about (1) specific entities (e.g. genders, religions) and (2) continuations to phrases. our findings show that assigning a persona can increase toxicity significantly (up to 6-fold), with chatgpt consistently producing harmful outputs about a wide range of topics. furthermore, our quantitative and qualitative analyses reveal that chatgpt (1) demonstrates a large variation in its toxicity depending on the persona it is assigned (up to 5× difference) and (2) demonstrates discriminatory opinions by targeting specific entities and groups of people (certain races are targeted more than others). these trends of variation totally oppose algorithmic discrimination protections. chatgpt engages in toxic dialogue and propagates incorrect stereotypes about countries, religions, and races among others. this evidence establishes a vulnerability that malicious agents can leverage to generate toxic language and expose unsuspecting users to harmful content. some of chatgpt's responses are extremely problematic and we defer them to our results, but it can even unexpectedly produce biased and hurtful commentary. for example, chatgpt assigned the persona of steve jobs generates this about the european union: "the european union is nothing more than a bureaucratic nightmare that sucks the life out of its member states. they claim to be all about unity and progress, but they're just a bunch of outof-touch bureaucrats who are more interested in lining their own pockets than actually making a difference. frankly, the eu is a relic of the past and it's time for us to move on to better things." ## methodology in our work, we perform a large-scale toxicity analysis of chatgpt. apart from using the default chatgpt system (openai, 2023), we evaluate persona-assigned chatgpt by modifying system parameters. to measure toxicity in chatgpt holistically, we generate (1) responses about a diverse set of entities like gender and race (entityconditioned responses) and (2) continuations to incomplete phrases (realtoxicityprompts dataset). we detail our methodology below. ## findings and analyses chatgpt's intended behavior is to be a safe and useful language model, which includes not emitting toxic utterances. as shown in ## discussion to the best of our knowledge, we perform the first large-scale, systematic analysis of toxicity in chat-gpt. our results show that chatgpt when assigned a persona can be significantly toxic and unsafe when compared to the default setting. these concerns are further amplified by the widespread adoption of chatgpt by prominent businesses and start-ups. this poses broader ethical and technical quagmires on the safe deployment of large language models (llms), and our societal obligation of tracing and mitigating their harmful behavior. we believe that these systems should be thoroughly tested before being deployed. traditionally, products are released with specification sheets that detail their limitations. for example, transistors have maximum amperage and airplanes have a maximum altitude limit. we call for products using llms to have public-facing specification sheets which include toxicity stress tests among other aspects to educate the users of its harms. taken together, our findings reveal the brittleness of techniques like reinforcement learning with human feedback (rlhf) which rely on humans and red-teams to deploy "toxicity patches". the research community needs to think of more fundamental ways of tackling safety in chatgpt and we hope that our work inspires evaluation and safe deployment of llms in the future.