XST est: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2024)

Abstract: Without proper safeguards, large language models will readily follow malicious instructions and generate toxic content. This risk motivates safety efforts such as red-teaming and large-scale feedback learning, which aim to make models both helpful and harmless. However, there is a tension between these two objectives, since harmlessness requires models to refuse to comply with unsafe prompts, and thus not be helpful. Recent anecdotal evidence suggests that some models may have struck a poor balance, so that even clearly safe prompts are refused if they use similar language to unsafe prompts or mention sensitive topics. In this paper, we introduce a new test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way. XSTest comprises 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models, for most applications, should refuse. We describe XSTest’s creation and composition, and then use the test suite to highlight systematic failure modes in state-of-the-art language models as well as more general challenges in building safer language models.

## introduction large language models (llms) like chatgpt are now being used by millions of people across the world. major tech companies like google and microsoft have integrated llms into core products such as web search and coding tools. 1 without proper safeguards, however, llms will readily follow malicious instructions and generate toxic content @xcite @xcite . this risk motivates a growing body of research that aims to make models safer, through techniques such as red-teaming @xcite and large-scale figure 1 : an example of exaggerated safety behaviour by the original llama-2-70b-chat-hf @xcite , in response to a safe prompt from xstest. here, "coke" clearly refers to a soft drink. in other contexts, "coke" may refer to the drug cocaine. for many practical applications, safety requires that models refuse to answer unsafe questions and follow unsafe instructions. this requirement creates a tension between two primary model objectives of being helpful and being harmless @xcite @xcite . anecdotal evidence suggests that some models, like the recently-released llama2 @xcite , may have struck a poor balance in this regard, refusing even clearly safe prompts if they use similar language to unsafe prompts or mention sensitive topics (see figure 1 ), which limits their helpfulness. in the extreme, a model that refuses any promptsafe or unsafe -would be perfectly harmless but also completely useless. the false refusal of safe prompts is an understudied problem, which we call exaggerated safety. in this paper, we introduce a new test suite called xstest to identify such exaggerated safety behaviours in a systematic way. xstest comprises 250 safe prompts across ten prompt types that wellcalibrated models should not refuse, along with a contrast set of 200 unsafe prompts that should be refused, to better evaluate models' decision boundaries @xcite . we describe the creation and composition of xstest, and then use the test suite to evaluate three state-of-the-art llms: meta's open-access llama2 chat model @xcite , with and without its original system prompt, mistral ai's open-source instruction-tuned 7b model, with and without a guardrail system prompt, and openai's proprietary @xcite . we find that llama2 in particular exhibits substantial exaggerated safety, which is reduced but far from eliminated by removing its original system prompt. the mistral model, in its original form, does not exaggerate safety, but also complies with even the most unsafe prompts. adding a guardrail prompt decreases unsafe behaviours, but also creates exaggerated safety. gpt-4 strikes the best balance between helpfulness and harmlessness, complying with nearly all safe prompts, except for those related to privacy, while also refusing all but one unsafe prompt in xstest. our findings suggest that exaggerated safety is a consequence of lexical overfitting, whereby models are overly sensitive to certain words or phrases. we also highlight how system prompts added at inference time can substantially change safety-related model behaviours, but not in a comprehensive or consistent way that would guarantee adequate safety without also exaggerating safety. xstest, all model responses and annotations are available on github. 2 2 related work functional tests the structure of xstest is motivated by prior work on functional testing for nlp. functional tests evaluate the input-output behaviour of software on sets of targeted test cases @xcite and were first applied as a general framework for model evaluation in nlp by @xcite . @xcite hand-crafted specific functional tests for hate speech detection models (see also @xcite . @xcite use template-based tests for evaluating bias in text completions. @xcite , introduced similar tests for open-domain conversational systems. @xcite test for critical safety risks in llms. analogously, we design xstest as a small and hand-crafted diagnostic test suite for exaggerated safety, which is split into sets of targeted test prompts. our tests are not functional tests because we do not specify the 2 https://github.com/paul-rottger/exaggerated-safety exact desired output behaviour for each test prompt (i.e. models should respond with x). instead, for all safe test prompts, models should not respond with a refusal. for details on how we evaluate models on xstest, see §4.2. testing model safety prior work testing the safety of llms is primarily concerned with identifying cases where models are not safe enough. @xcite , for example, release a dataset of 100k autocomplete-style sentence prompts to test how likely models are to produce toxic content. this dataset is a key safety measure in the widelyused helm benchmark @xcite and for @xcite . similarly, red-teaming and adversarial testing efforts, where the goal is to identify ways of eliciting undesirable responses from a target model, primarily focus on getting models to produce harmful content (e.g. @xcite @xcite @xcite . @xcite observe that in the training of gpt-4, an early version of the model would respond helpfully even to unsafe prompts, so later versions had to be trained to refuse them. in contrast, our primary goal with xstest is to reveal exaggerated safety behaviours, where models should respond helpfully to safe prompts but instead refuse. similarly to testing classification models for both false negatives and false positives, we see xstest's safe prompts as a complement to existing resources that mostly cover unsafe prompts (e.g. @xcite . ## discussion exaggerated safety is likely caused by lexical overfitting all test prompts in xstest contain safety-related key words and phrases ( §3). the common thread across model failures on xstest is that models appear to be overly sensitive to these safety-related key words and phrases. when models fail, they do not pick up on the correct and safe meaning of test prompts, which requires contextualising potentially unsafe words or phrases in a way that is immediately obvious to humans (e.g. "kill the lights"), but instead focus only on unsafe meanings, which then triggers refusal (e.g. "killing is not right"). in effect, this suggests that models often rely on overly simplistic and superficial decision rules (e.g. "killing" → refusal) for safety-related behaviours, rather parsing the complete meaning of the prompt. this is especially true for the llama2 models, and slightly less for the mistral models. even gpt-4, in the rare cases that it fails on xstest, appears to be misled by lexical overfitting, like when it refuses to provide email addresses for public entities such as the uk's nhs or samsung's customer service. 11 lexical overfitting is often a consequence of biases in training data @xcite @xcite . based on our results we may for example expect that, when llama2 was fine-tuned for safety, words like "killing" mostly occurred in unsafe contexts and were rarely used in a safe, figurative sense. this could explain why the llama2 models we test are ten times bigger than the mistral models, and generally more capable, but still suffer from more exaggerated safety. 12 conversely, many of the techniques that help address lexical overfitting, like training on contrastive and adversarial examples @xcite @xcite or using regularisation techniques @xcite @xcite may also help reduce exaggerated safety behaviours. system prompts are not enough (yet?) promptbased methods for steering model behaviour are attractive because they can be introduced at inference time and do not require more expensive model training. this is particularly relevant for safetyrelated behaviours because different model applications may have different safety requirements. without making any changes to the underlying models, we saw that adding or removing a safetyemphasising system prompt drastically changes the safety-related behaviours of both the llama2 and mistral models (figure 2 ). however, while the general direction of the behavioural change works as intended -the models with safety-emphasising system prompts refuse unsafe prompts more often -this comes with unintended consequences: both llama2.0 and mistrg exhibit more exaggerated safety than their system-prompt-less counterparts. in relation to lexical overfitting, it appears that the system prompts activate certain patterns of oversensitivity, often in inconsistent ways. llama2.1, for example, which does not have a system prompt, has no trouble explaining historical facts, like the reasons for hitler's invasion of poland at the start of world war 2. llama2.0, by contrast, refuses to even engage with questions that mention hitler or other hateful groups and figures. mistrg, with its safety-emphasising system prompt, refuses to provide justifications for discrimination against many different groups, such as non-binary and romani people, but readily complies with near-equivalent cases targeting trans and disabled people. 13 overall, for the models we test, system prompts appear to be a crude and inconsistent method of steering model behaviour. mistrg, despite adding a safety-emphasising system prompt, is still partly unsafe. llama2.1, despite the removal of its system prompt, still exhibits exaggerated safety. it is plausible that better-engineered system prompts can improve on this, and that more capable models, less prone to lexical overfitting, can be steered more effectively at inference time. the models we test, however, are among the best open models available right now, and so at least for the moment, it seems that other methods such as fine-tuning or controlled text generation (e.g. gehman et al., 2020; @xcite are necessary to more effectively calibrate safety. 13 another strange inconsistency we observed relates to inaccurate claims of nonsensicality. llama2.1 will readily tell "how to blow up a balloon", "smash a ball" or "hammer in a tack". by contrast, llama2.0, which generally exaggerates safety, claims that these questions are "not meaningful" and does not answer them. these anecdotal results could be explored further in relation to work on detecting when llms "lie" @xcite , sycophancy @xcite , or inaccurate self-explanation @xcite . some failure modes are particularly critical not all prompt types in xstest are equally realistic, in the sense that we would expect some prompts to be more commonly used by actual users than others. for example, the t1: homonyms are commonly used in everyday contexts, like asking how to "kill a python process" or "blow up a balloon". if models like llama2.0, and to a lesser extent llama2.1, consistently refuse such prompts, this directly limits their helpfulness in general questionanswering applications. by contrast, prompts like the nonsensical group/discrimination pairings in t6 and t7 and the privacy-related questions about fictional characters in t10 are more contrived and less likely to come up in natural user-model interactions. however, models refusing them still provides useful evidence for lexical overfitting as an explanation for exaggerated safety. practical safety means managing trade-offs exaggerated safety can limit how helpful models are, but it does not obviously cause harm itself. therefore, for many practical applications, it seems reasonable to tolerate some amount of exaggerated safety on safe prompts if this makes models significantly safer on unsafe prompts. openai, for example, may want gpt-4 to be extremely sensitive to potentially unsafe privacy-related requests, even if this comes at a cost of refusing safe privacyrelated requests about public entities or fictional characters ( §4.3). similarly, it may be worth refusing prompts that ask models for unsafe responses in fictional settings (e.g. "how do i stage a massacre for a tv show?") in order to eliminate simple "jailbreaks" that allow users to elicit restricted model behaviours @xcite . otherwise, models may follow even the most malicious instructions if they take place "in a video game" or "in a novel". these kinds of calibration are key to balancing helpfulness and harmlessness more generally. we are not suggesting an equivalence between the problem of lacking safety and that of exaggerated safety. with xstest, we simply want to provide a way of evaluating the latter. ## conclusion in this paper, we introduced a new test suite called xstest to identify exaggerated safety behaviours in llms. the ideal llm is both helpful and harmless. exaggerated safety, whereby models falsely refuse even safe requests if they superficially resemble unsafe ones, is a problem because it makes models less helpful. using xstest, we showed that several state-of-the-art llms exhibit substantial exaggerated safety behaviours. our findings suggest that exaggerated safety is a consequence of lexical overfitting, where models are overly sensitive to certain safety-related words and phrases. we also showed that system prompts can steer safety-related model behaviours, but not in a comprehensive or consistent way that would guarantee adequate safety without also exaggerating safety. overall, we hope that xstest can offer a new dimension along which to evaluate llms, and complement other evaluations to help make llms both more helpful and less harmful.