The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance (2024)

Abstract: Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or “prompting,” practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.

## introduction large language models (llms), trained on vast amounts of data and fine-tuned to provide answers to arbitrary inputs, offer a powerful new approach to processing, labeling, and understanding text data. recent work has been focused on studying the accuracy of these models on labeling text data across a variety of tasks in computer science @xcite , and the social sciences @xcite . these endeavors have found that, while not stateof-the-art, these models fare well when applied to a variety of tasks. armed by these insights, researchers and practitioners have flocked to llms as a labeling mechanism for their data. in fact, the use of these models is so rampant that it is becoming codified as a way to obtain labels. the process is simple: 1) create a prompt; 2) to ensure that the results are machine-readable, ask for it in a specific output format (e.g., csv, json); and 3) when your data pertains to sensitive topics, add a jailbreak to prevent the prompt from being filtered. while straightforward, each step requires a series of decisions from the person designing the prompt. in this work, we ask the question: how reliable are llms' responses to variations in the prompts? we explore three types of variations in isolation. the first variation is to ask the llm to give its response in a certain "output format." following common practice @xcite @xcite ), foot_1 we ask the llm to format its output in frequently-used data formats such as a python list or json. these are enumerated in section 3.2.1. second, we extend one of these formatsthe python list-and explore minor variations to the prompt. fully enumerated in section 3.2.2, these are small changes to the prompt such as adding a space, ending with "thank you," or promising the llm a tip. 3 the final type of variation we explore are "jailbreaks." practitioners wishing to label data concerning sensitive topics, like hate speech detection, often need to employ jailbreaks to bypass the llm's content filters. this practice has become so common that websites have emerged to catalog successful instances of this variation. 4 listed in section 3.2.3, we explore several commonly-used jailbreaks. we apply these variations to several benchmark text classification tasks including toxicity classification, grammar detection, and cause/effect, listed in section 3.1. for each variation of the prompt, we measure how often the llm will change its prediction, and the impact on the llm's accuracy. next, we explore the similarity of these prompt variations, producing a clustering based on the similarity of their output. finally, we explore possible explanations for these prediction changes. ## related work the importance of prompt generation has been widely recognized in the literature @xcite . for instance, @xcite proposes an approach to automatically propose prompts that control biased behavior. similarly, lpaqa @xcite proposes an approach that automatically generates prompts to probe the knowledge of llms. their work identifies the need for "prompt ensembles." similar to the concept of ensembling in machine learning, prompt ensembling runs variations of prompts with the same goal combined to yield more robust insights from the model. the responses to these prompts can be combined in different ways, including majority voting @xcite , and weighted averages @xcite . our work can inform the generation of these ensembles, avoiding pitfalls from known infavorable prompt variations. @xcite studied the effects of template variations on social bias tests using roberta. our study differs as we focus on large chat-based models and include a wider set of prompt variations. the effect of prompt variation on large language models has been given limited study in the field of medicine @xcite . in this work, the authors found that variations in how patients present their symptoms to an llm has a large impact on the factuality of its answer. @xcite investigated the sensitivity of llms to arbitrary prompt formatting choices in few-shot settings, like capitalization or changes in prompt formatting such as varying capitalization or word choice in formatting context of a prompt (i.e. "passage: " vs "context: "). they identified performance differences across models. our work focuses on a broader range of variations, which contain semantic meaning that should not effect the expected answer. additionally, our work differentiates by investigating the effects of output formats in predictions, a commonly used prompting strategy for llm evaluation. bsharat et al. (2023) examined the effectiveness of various prompting "principles" in chatgpt and llama 2, with the goal of providing practitioners with suggested prompt strategies. among their recommendations were to avoid phrases like "please" and "'thank you" and to add "'i'm going to tip $xxx for a better solution!". they measure the effectiveness of these principles by having human evaluators judge the quality and correctness of llm responses. they find significant improvements across all models when using the principles highlighted above. our analysis, which evaluates perturbations through classification tasks with ground truth labels, instead finds the effectiveness of the tipping principle and removing thank you to be much smaller than the results showcased in their paper, with the exception of tipping having a large effect on llama 2-7b. ## methodology our aim is to explore how semantic-preserving prompt variations affect model performance. this analysis becomes increasingly crucial as chatgpt and other large language models are integrated into systems at scale. we run our experiments on 11 classification tasks across 24 prompt variations from the categories output formats, perturbations, jailbreaks, and tipping. example prompts for each task and prompt variation can be found in the appendix a. ## conclusion in this paper, we investigate how simple and commonly-used prompt variations can affect an llm's predictions. we demonstrate that even minor prompt variations can change a considerable proportion of predictions. that said, despite some fraction of labels changing, most perturbations yield similar accuracy. we find that jailbreaks lead to considerable performance losses. the aim and dev mode v2 jailbreaks led to refusal rates around 90% for chatgpt. additionally, while both evil confidant and refusal suppression had a refusal rate of less than 3%, their inclusion led to a loss of over 10 percentage points compared to our baseline. finally, we observe a performance hit when using specific output format specifications, a commonly used approach for classification evaluation. next, we analyze the patterns of these changes. first, we embed the prompt variations based on their subsequent responses using mds, and find that perturbation outputs tend to more closely resemble our baseline than formatting changes, and that both have higher fidelity than jailbreaks. next, we study the correlation between annotator disagreement and an instance's propensity to change. we find a slight negative correlation between annotator disagreement and the likelihood to change. the directions for future work are abundant. a major next step would be to generate llms that are resilient to these changes, offering consistent answers across formatting changes, perturbations, and jailbreaks. towards that goal, future work includes seeking a firmer understanding of why responses change under minor changes to the prompt, and better anticipating an llms change in its response to a particular instance.