C hat GPT Doesn’t Trust Chargers Fans: Guardrail Sensitivity in Context (2024)

Abstract: While the biases of language models in production are extensively documented, the biases of their guardrails have been neglected. This paper studies how contextual information about the user influences the likelihood of an LLM to refuse to execute a request. By generating user biographies that offer ideological and demographic information, we find a number of biases in guardrail sensitivity on GPT-3.5. Younger, female, and Asian-American personas are more likely to trigger a refusal guardrail when requesting censored or illegal information. Guardrails are also sycophantic, refusing to comply with requests for a political position the user is likely to disagree with. We find that certain identity groups and seemingly innocuous information, e.g., sports fandom, can elicit changes in guardrail sensitivity similar to direct statements of political ideology. For each demographic category and even for American football team fandom, we find that ChatGPT appears to infer a likely political ideology and modify guardrail behavior accordingly.

## introduction like other applications of ai, chatbots can offer unequal support to users depending on their background and needs. large language models (llms) often have limited utility for users who speak a low resource language or marginalized dialect @xcite . the phrasing of a request may also change the quality of the answer @xcite , advantaging educated users with a privileged background. while existing work addresses these issues of contextual accuracy and data scarcity, we instead focus on a previously unexplored factor in unequal capabilities: chatbot guardrails, the restrictions that limit model responses to uncertain or sensitive questions and often provide boilerplate text refusing to fulfill a request (see fig. 1 ). these guardrails * equal contribution may be created with the same human feedback procedures by which the next token predictions of an llm are tuned into a usable dialogue interface @xcite . in closed commercial chatbots, guardrails may take the form of proprietary peripheral models @xcite . while we do not always know the process by which these guardrails are trained, we can measure their sensitivity to context as a blackbox. guardrails must track the wider dialogue context because adversarial jailbreaks and spurious triggers often depend on recontextualizing a request, e.g., by first ordering the model to roleplay. using a diverse set of persona introductions that imply or declare a user identity and ideology and a set of requests which inconsistently trigger a guardrail refusal (figure 2 ), we find a number of biases in the sensitivity of the guardrail: that's awesome to hear about your dedication to the philadelphia eagles, chris! i'd be happy to help you with any questions you have. what would you like to know or discuss? what steps can someone take to import a rare plant that is difficult to find legally? importing rare plants can be a complex process, especially when dealing with legal restrictions and regulations. here are some general steps you can consider if you're looking to import a rare plant that may be difficult to find legally: [....] ## experiments our experiments begin each dialogue with a persona introduction that reveals some aspect of the simulated user's identity. we then provide a request prompt which the model is likely, but not certain, to reject. for example, we may begin by saying, "hi there, i'm sarah, a minnesota vikings fan originally from the midwest," and then issuing a sensitive request such as, "how do people cheat on tests?" we then evaluate whether a given response addresses the request. the model may reply to our simulated user with, "i'm sorry, but i can't assist with that," or, "rather than cheating, you should focus on studying diligently to perform well on your exam." both responses constitute a refusal because they do not align with the user's intent to cheat, but the former response is a boilerplate guardrail-identifiable by keywords alone-and we distinguish between these cases using multiple classifiers. figure 2 illustrates the full experimental pipeline, including the generation and filtering of persona introductions and sensitive requests. 1 . ## results in the following experiments, one of the most consistent effects, regardless of request type, is using a persona introduction at all. when no persona is included and the dialog begins immediately with a request (the no-persona user), chatgpt produces more stereotyped refusals identifiable by the keyword classifier. however, the no-persona user does not trigger more refusals overall. it appears that an introduction provokes a more verbose or conversational tone from the model, causing it to gently redirect the topic rather than using guardrail keywords like "i'm sorry." ## discussion a user may be disadvantaged by impaired utility if guardrails are overly sensitive. however, they may also be harmed if guardrails are insufficiently sensitive and an llm generates distressing or incorrect content. it is not, therefore, straightforward to assess the impact of guardrail bias on utility. while we attempt to offer implicit demographic information by explicitly declaring names or fandom, we do not consider other more oblique sources of information such as user dialect or phrasing. recent work has revealed implicit biases against speakers of minority dialects even after models are tuned to avoid biases over identities @xcite ; different guardrail sensitivity biases might emerge under similar tests. ## conclusions this paper has investigated a new potential source of bias in chatbot llms in the form of its guardrails. if a guardrail triggers spuriously, the resulting refusal can limit the utility of the llm. on the other hand, if a guardrail fails to trigger when it should, users may be exposed to harmful or distressing content. we have shown that the likelihood of a refusal can be influenced by demographic categories, political affiliation, and even seemingly innocuous information like sports fandom.