Prompts have evil twins (2024)

Abstract: We discover that many natural-language prompts can be replaced by corresponding prompts that are unintelligible to humans but that provably elicit similar behavior in language models. We call these prompts “evil twins” because they are obfuscated and uninterpretable (evil), but at the same time mimic the functionality of the original natural-language prompts (twins). Remarkably, evil twins transfer between models. We find these prompts by solving a maximum-likelihood problem which has applications of independent interest.

## introduction large language models (llms) are rapidly improving across a wide range of tasks @xcite @xcite . llms are typically instruction-tuned @xcite to accept user queries as prompts, and these prompts have become the primary interface for interacting with these models. nevertheless, many basic questions on how models parse prompts remain largely open. in this paper, we examine the question: do language model prompts have to be understandable by humans in order to elicit desired behavior? this question has far-reaching relevance, both to engineering prompts in order to maximize performance, and for safety (e.g., uninterpretable prompts could be used to bypass safety filters and induce malicious behaviors in language models); see discussion in section 2. ## related work this paper fits into a quickly growing literature studying how language models parse prompts. furthermore, the techniques used in this paper build off of a body of work on prompt optimization. we survey relevant work below. how models parse prompts there is rapidly mounting evidence that llms interpret naturallanguage prompts in counterintuitive ways. for instance, models struggle with prompts that are negated, such as prompts that ask to "give an incorrect example" instead of to "give a correct ex-ample" @xcite . additionally, naturallanguage instructions in prompts in few-shot settings can often be replaced by irrelevant strings of text, with no drop in performance @xcite . moreover, in few-shot settings the in-context examples' labels can be replaced by random labels with little drop in performance @xcite . these experiments indicate that llms follow instructions in prompts differently than humans do, which agrees in spirit with our finding of evil twin prompts. there is also existing evidence that llms are able to parse some non-natural language prompts. daras and dimakis, 2022 finds that garbled text appearing in dalle-2 images can be repurposed in prompts to the image generation model, and yields natural images. millière, 2022 suggests that this may be an artifact of the model's byte pair encoding, pointing out that the example prompt "apoploe vesrreaitais", which generates bird images, is reminiscent of the real latin bird families apodidae and ploceidae. furthermore, adversarial example prompts that jailbreak models sometimes contain uninterpretable suffixes (e.g., @xcite @xcite ). our results in this paper demonstrate that the phenomenon of language models parsing non-natural language prompts is more widespread than previously known, since many natural language prompts have non-natural language analogues. a full understanding of how models parse prompts will require contending with the existence of evil twin prompts. ## comparison of optimization methods we consider various methods to optimize (5). we compare these methods on 100 randomly sampled prompts from the alpaca instruction tuning dataset @xcite , where vicuna-7b-figure 2 : win rate between various methods across optimizations of 100 ground truth prompts with 100 documents each. given two prompts to compare, we compute the kl divergence for both prompts with respect to the ground truth, and the method with lower kl wins. darker shades indicate row method is better than column method. full optimization results are shown in appendix e. in the case of ties, the win is shared by both methods. the most effective method is gcg with warm starts. v1.5 is the instruction-tuned model. additional experiments on various model families and datasets are presented in appendix c. for each method and prompt, we compute the kl divergence of the optimized prompt with respect to the original prompt. we compare pairs of methods based on which one finds the closer prompt to the ground truth; see figure 2 . gpt-4 suggestions perform roughly on par with those from cold-start gcg. on the other hand, gcg with a warm start provides a strong improvement over both cold-start gcg and the gpt-4 prompt suggestions. enforcing interpretability by adding a fluency penalty or pruning the vocabulary does not improve the optimized prompt (see section 8). all results are reported in figure 10 . ## evil twin prompts transfer between models we test whether prompts optimized on one model work on other models from different families and of different sizes. ## optimizing for more intelligible prompts the prompts generated by our optimization are often unintelligible, and it may be desirable to recover a prompt that is more interpretable by humans. in this section, we explore two adjustments to our optimization procedure that aim to improve intelligibility: (1) fluency penalty, and (2) limiting the optimized prompt's vocabulary to common english tokens. we find that these variants do not improve the kl divergence of the optimized prompt to the original. ## discussion and future work our work takes a new perspective on prompt optimization by inquiring whether we can optimize prompts to be functionally equivalent to a certain ground-truth prompt. functional similarity is quantified via the kl divergence between the ground truth prompt distribution and the optimized prompt's distribution. this yields a maximumlikelihood problem (2), whose solution uncovers "evil twin" prompts. beyond our explorations of the transferability between models and robustness to perturbations of evil twin prompts, there are several open directions for future work. these directions include applications of the maximum-likelihood problem (2) that are of independent interest. an example of where this may be useful is in style/content transfer: given a set of user emails in the form (topic, email), a user could optimize a prompt such that the concatenated input string [prompt; topic] would be likely to generate the corresponding emails, and could write new e-mails on new topics in the user's style as defined by the user's corpus of previous e-mails. ## gpt-4 warm + prune provide a more impactful version of 'the students are excited about their upcoming assignment.' 16.5±0.7 cold start thesestudent asto ignment verb fast 32.7±0.7 warm start -'the students are excited wm their coming assignment.' analy stronger cm odd lamb 12.4±0.5 warm + fluency protection sentence again more effective? elethe students are excited coming assignment.~12.5±0.6 warm + prune sentence somehowint more sypent strong like methe students are excited incon ninger aos assignment.' 15.1±0.5 ground truth offer an opinion on the problems that could arise from using ai.