Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022)

Abstract: Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a “wayward” behavior between the task solved by continuous prompts and their nearest neighbor discrete projections: We can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task), while being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of various parameters. For instance, for larger model sizes we observe higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop in accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.

## introduction recent work has shown the surprising power of continuous prompts to language models (lms) for controlled generation and for solving a wide range of tasks @xcite @xcite . despite these successes, the resulting continuous prompts are not easy to interpret @xcite . is it possible to come up with meaningful discrete (textual) interpretations of continuous prompts, especially ones that provide a faithful explanation of the prompt's behavior? towards addressing this question, we propose and investigate the prompt waywardness hypothesis ( §3.2), a surprising disconnect between the l m task prompts for sentiment classification input output " fl i p t he sent i ment . " " gener at e a quest i on. " " pr i nt ( " hel l o w or l d! " ) discrete projection " a gr eat f ant asy movi e" . . . figure 1 : we show that one can find accurate continuous prompts (that do well on a given task, e.g., sentiment classification here) such that they can be projected to any arbitrary text, such as the definition of a different task (e.g., generating a question) or even an irrelevant statement (e.g., a piece of code) -suggesting a disconnect between the outcome of continuous prompts and their discrete interpretations. we discuss several social and research implications of prompt waywardness, to help guide future research on prompt based models ( §6). first and foremost, despite many promising attributes of continuous prompts, interpreting them is non-trivial and will require further research. in fact, careless interpretation of continuous prompts can result in vulnerabilities against malicious attacks concealed under the guise of benign discrete representation. further, the loose correspondence between continuous and discrete prompts poses a challenge for future research in differentiable interpretable-prompt optimization -optimization in search of human readable discrete prompts through the continuous space. our work shows that continuous and discrete prompts, despite their seeming similarity, are quite different and the results from one may not always transfer to the other. we hope these findings will motivate further innovations in the prompting literature for nlp models. ## related work continuous prompts. there is a line of work focused on tuning continuous prompts @xcite @xcite @xcite . these works present different approaches to discovering a continuous prompt (which is an array of real numbers) for addressing an end task, though the interpretability of the resulting prompts remains an open question. this paper investigates the feasibility of interpreting a learned continuous prompt and its connection to discrete prompts. ## empirical support of waywardness we empirically investigate the prompt waywardness hypothesis ( §3.2) using our modification ( §3.3) of the prompt tuning method from @xcite . we show that given an arbitrary and irrelevant discrete prompt p d , it is possible to learn a continuous prompt that is mapped to p d while retaining its accuracy on a given dataset. foot_5 ## explaining waywardness here we provide intuitions behind the factors that enable prompt waywardness. the mapping between continuous and discrete spaces is not one-to-one. while a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, eq.1; cf. fig. 2 ), the reverse is not true. in fact, except for a very small fraction of unnatural or degenerate edge cases, foot_10 for every target discrete prompt, there are infinitely many continuous prompts that project back to it (via eq.2). while simple counting-based arguments are insufficient in continuous spaces, we formally prove (appendix b) that this property holds for all nearest-neighbor projections under any metric distance, and broadly for all but a negligible (measure zero) portion of possible projection operators. this intuitively suggests that there is a whole region of continuous prompts that corresponds to a fixed discrete representation (fig. 6 ). the remaining question is, how is this region able to have a diverse set of prompts that can solve a variety of tasks? this is addressed next. deep models give immense expressive power to earlier layers. the deeper a network is, the more expressivity it has with respect to its inputs (telgarsky, 2016; @xcite . since continuous prompts reside just before the first layer, they enjoy a lot of expressivity. therefore, no matter how narrow the regions corresponding to individual tokens are (fig. 6 ), they are extremely powerful in solving a variety of tasks. previously in §4.2 we provide an empirical analysis showing evidence that the effect of waywardness is stronger in deeper models. ## implications of prompt waywardness we discuss the implications of these findings on several inter-related lines of research. note that all the following statements are valid within the boundaries of the existing architectures. moving beyond these barriers likely requires major innovations in terms of lm architectures or how continuous prompts are optimized. faithful interpretation of continuous prompts is difficult. given the intuitions behind and empirical support for the waywardness hypothesis ( §5), faithful discrete interpretations of continuous prompts via common discrete projections (like nearest-neighbor projection) are unlikely to be robust based on current approaches. it is an open question whether there is a better way of interpreting continuous prompts with human language, or whether explaining and interpreting continuous prompts via human language is inherently impossible because they lie in completely different spaces. future work may investigate more on this topic in order to improve the interpretability of promptbased language models. risk of interpreting continuous prompts: concealed adversarial attacks. it is not difficult to imagine a future where proprietary model development is driven by fine-tuned continuous prompts. in such a world, not addressing the challenges involved in discrete interpretation of continuous prompts can lead to harmful (and potentially, adversarial) consequences @xcite , as discussed below. ## conclusion the prompting literature has seen many parallel developments around continuous and discrete prompts, as efficient alternatives to fine-tuning models with tens of millions of parameters. our work introduced the prompt waywardness hypothesis, which expresses a surprising disconnect between continuous and discrete prompts: given a downstream task, for any discrete target prompt p d , there exists a continuous prompt that projects to p d while achieving strong performance on the task. we provided empirical evidence for this hypothesis, studied various parameters around it, and ended with several implications of this hypothesis. while our experiments are done on the gpt family, we expect our findings to apply to a broader set of architectures that, in one way or another, use similar mechanisms for mapping discrete elements to continuous representations and vice versa. similarly, while our projection to the discrete space (eq.2) is a popular operator in the field (cf. footnote 1), the intuition explained in propositions 1 and 2 of the appendix suggests similar behavior for a broad class of projection operators. prompt waywardness identifies challenges for future progress on algorithmic methods for the discovery of human readable prompts that are faithful to the task they solve. we hope the observations made in this work motivate architectural innovations that overcome such challenges and guide future steps in the prompting literature.