Original: P ROMPT W AYWARDNESS : The Curious Case of Discretized Interpretation of Continuous Prompts
Revised: P ROMPT W AYWARDNESS : The Curious Case of Discretized Interpretation of Continuous Prompts

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Fine-tuning continuous prompts for targettasks has recently emerged as a compact alternative to full model ﬁne-tuning.
Revised: Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model ﬁne-tuning.

Original: Motivated bythese promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithfulto the problem they solve.
Revised: Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve.

Original: In practice, we observe a “ wayward ” behavior between the tasksolved by continuous prompts and the nearestneighbor discrete projections of these prompts:
Revised: In practice, we observe a “ wayward ” behavior between the task solved by continuous prompts and the nearest neighbor discrete projections of these prompts:

Original: One can ﬁnd continuous prompts that solve atask while being projected to an arbitrary text(e.g., deﬁnition of a different or even a contradictory task) and simultaneously being withina very small (2%) margin of the best continuous prompt of the same size for the task.
Revised: One can ﬁnd continuous prompts that solve a task while being projected to an arbitrary text (e.g., deﬁnition of a different or even a contradictory task) and simultaneously being within a very small (2%) margin of the best continuous prompt of the same size for the task.

Original: We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of designchoices.
Revised: We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of design choices.

Original: For instance, larger models exhibithigher waywardness, i.e, we can ﬁnd promptsthat more closely map to any arbitrary textwith a smaller drop in accuracy.
Revised: For instance, larger models exhibit higher waywardness, i.e, we can ﬁnd prompts that more closely map to any arbitrary text with a smaller drop of accuracy.

Original: These ﬁndings have important implications relating tothe difﬁculty of faithfully interpreting continuous prompts and their generalization acrossmodels and tasks, providing guidance for future progress in prompting language models.
Revised: These ﬁndings have important implications relating to the difﬁculty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.

Original:  Introduction
Revised:  Introduction

Original: Recent work has shown the surprising power ofcontinuous prompts to language models (LMs) forcontrolled generation and for solving a wide rangeof tasks (Li and Liang, 2021; Lester et al., 2021;
Revised: Recent work has shown the surprising power of continuous prompts to language models (LMs) for controlled generation and for solving a wide range of tasks (Li and Liang, 2021; Lester et al., 2021;

Original: Min et al., 2021).
Revised: Min et al., 2022).

Original: Despite these successes, theresulting continuous prompts are not easy to interpret (Shin et al., 2020).
Revised: Despite these successes, the resulting continuous prompts are not easy to interpret (Shin et al., 2020).

Original: Is it possible to come upwith meaningful discrete (textual) interpretationsof continuous prompts, especially ones that providea faithful explanation of the prompt’s behavior?
Revised: Is it possible to come up with meaningful discrete (textual) interpretations of continuous prompts, especially ones that provide a faithful explanation of the prompt’s behavior?

Original: Towards addressing this question, we proposeand investigate the Prompt Waywardness hypothesis (§3.2), a surprising disconnect between the
Revised: Towards addressing this question, we propose and investigate the Prompt Waywardness hypothesis (§3.2), a surprising disconnect between the

Original: L M task prompts for sentiment classification
Revised: L M task prompts for sentiment classification

Original: Input
Revised: Input

Original: Output "
Revised: Output "

Original: Fl i p t he sent i ment . "
Revised: Fl i p t he sent i ment . "

Original: " Gener at e a quest i on. "
Revised: "Gener at e a quest i on. "

Original: " pr i nt ( " Hel l o Wor l d! " ) discrete projection " A gr eat f ant asy movi e" . . .
Revised: "pr i nt ( "Hel l o Wor l d! ") discrete projection "A gr eat f ant asy movi e" . . .

Original: intended behavior of continuous prompts and their[Figure] nearest-neighbor discrete (language) representa- [Figure]
Revised: intended behavior of continuous prompts and their nearest-neighbor discrete (language) representations.

Original: tions. 1 In particular, we show that one can ﬁnd con- [Figure] tinuous prompts that perform a desired task while, 
Revised: 1 In particular, we show that one can ﬁnd continuous prompts that perform a desired task while, at the same time, project to any given target text.

Original: [Figure]

Original: at the same time, project to any given target text.

Original: [Figure]

Original: This indicates that there is little correspondence[Figure] between continuous prompts and their discrete in- [Figure]
Revised: This indicates that there is little correspondence between continuous prompts and their discrete interpretation.

Original: terpretation. For instance, a continuous prompt that[Figure] effectively solves the sentiment classiﬁcation task 
Revised: For instance, a continuous prompt that effectively solves the sentiment classiﬁcation task in Fig.1, when projected onto discrete space, might appear as the deﬁnition of a different task ( “ﬂip the sentiment” ).

Original: [Figure] in Fig.1, when projected onto discrete space, might [Figure]
Revised: Intuitively, continuous prompt optimization is a highly non-convex problem with numerous local minima.

Original: appear as the deﬁnition of  a different task ( “ﬂip the [Figure] sentiment” ). [Figure]
Revised: More surprisingly, many of these prompts (e.g., those near embedded text) are very effective at solving a desired task.

Original: We conduct extensive analysis showing Way- [Figure] wardness on ﬁve classiﬁcation datasets (§4).Empir-
Revised: We conduct extensive analyses showing Waywardness on ﬁve classiﬁcation datasets (§4).

Original: [Figure] ically, we ﬁnd the existence of wayward prompts —[Figure] prompts that solve each of these tasks while having arbitrary natural language projections.
Revised: Empirically, we ﬁnd the existence of wayward prompts — prompts that solve each of these tasks while pro- jecting to arbitrary natural language text.

Original: To study a variety of projected text, we experiment with 60+ sentences, eithera discrete prompt from another task (from Mishra et al. 2021b) ora random sen- tence from a large text corpus.
Revised: To study a variety of projected text, we experiment with 60+ sentences, either discrete prompts from other tasks (from Mishra et al. 2022b) or random sentences from a large text corpus.

Original: We observe that it ispossible to ﬁnd prompts that project to a given discrete prompt (token overlap 94% F1) while scoringwithin 2% accuracy of the best continuous promptsbased solution for the task.
Revised: We observe that it is possible to ﬁnd prompts that project to a given discrete prompt (token overlap 94% F1) while scoring within 2% accuracy of the best continuous promptsbased solution for the task.

Original: Further analysis showsthat the effect of Waywardness gets worse for largermodels and longer prompts.
Revised: Further analysis shows that the effect of Waywardness gets worse for larger models and longer prompts.

Original: We explain this surprising behavior by relating it to several structuralproperties of large language models (§5).
Revised: We explain this surprising behavior by relating it to several structural properties of large language models (§5).

Original: We discuss several social and research implications of prompt waywardness, to help guide futureresearch on prompt based models (§6).
Revised: We discuss several social and research implications of prompt waywardness, to help guide future research on prompt based models (§6).

Original: First andforemost, despite many promising attributes of continuous prompts, interpreting them is non-trivialand will require further research.
Revised: First and foremost, despite many promising attributes of continuous prompts, interpreting them is non-trivial and will require further research.

Original: In fact, carelessinterpretation of continuous prompts can result invulnerabilities against malicious attacks concealedunder the guise of benign discrete representation.
Revised: In fact, careless interpretation of continuous prompts can result in vulnerabilities against malicious attacks concealed under the guise of benign discrete representation.

Original: Further, the loose correspondence between continuous and discrete prompts poses a challenge for future research in differentiable interpretable-promptoptimization – optimization in search of humanreadable discrete prompts through the continuousspace.
Revised: Further, the loose correspondence between continuous and discrete prompts poses a challenge for future research in differentiable interpretable-prompt optimization – optimization in search of human readable discrete prompts through the continuous space.

Original: Our work shows that continuous and discrete prompts, despite their seeming similarity, arequite different and the results from one may notalways transfer to the other.
Revised: Our work shows that continuous and discrete prompts, despite their seeming similarity, are quite different and the results from one may not always transfer to the other.

Original: We hope these ﬁndingswill motivate further innovations in the promptingliterature for NLP models.
Revised: We hope these ﬁndings will motivate further innovations in the prompting literature for NLP models.

Original:  Related Work
Revised:  Related Work

Original: Continuous prompts.
Revised: Continuous prompts.

Original: There is a line of work focused on tuning continuous prompts (Li and Liang,2021; Lester et al., 2021; Zhong et al., 2021; Qinand Eisner, 2021;
Revised: There is a line of work focused on tuning continuous prompts (Li and Liang, 2021; Lester et al., 2021; Zhong et al., 2021; Qin and Eisner, 2021;

Original: Zhou et al.,  2021).
Revised: Zhou et al., 2021; Zhong et al., 2021).

Original: A recurringtheme in this line of work is  the strength of continuous prompt in results in strong, yet compactmodels—compared to conventional architectureﬁne-tuning approaches.
Revised: These works present different approaches to discovering a continuous prompt (which is an array of real numbers) for addressing an end task, though the interpretability of the resulting prompts remains an open question.

Original: Motivated by the successof continuous prompt tuning, this paper investigates the feasibility of interpreting a learned continuous prompt and its connection to discrete prompts.
Revised: This paper investigates the feasibility of interpreting a learned continuous prompt and its connection to discrete prompts.

Original: Discrete prompts.
Revised: Discrete prompts.

Original: The release of GPT-3 (Brownet al., 2020) sparked a lot of excitement about theemergent ability of LMs in following discrete natural language prompts.
Revised: The release of GPT-3 (Brown et al., 2020) initiated a body of work on the emergent ability of LMs to follow discrete natural language prompts.

Original: Consequently, countlessfollow-up studies have used manually-designed discrete prompts for probing LMs (Petroni et al., 2019;
Revised: Consequently, several followup studies have used manually-designed discrete prompts for probing LMs (Petroni et al., 2019;

Original: Jiang et al., 2020), improving LMs few-shot ability (Schick and Schütze, 2021; Gao et al., 2021;
Revised: Jiang et al., 2020), improving LMs’ few-shot ability (Schick and Schütze, 2021; Gao et al., 2021;

Original: Le Scao and Rush, 2021), and their zero-shot ability as well as transferability (Mishra et al., 2021a; Reynolds and McDonell, 2021).
Revised: Le Scao and Rush, 2021), and their zero-shot ability as well as transferability (Mishra et al., 2022a; Reynolds and McDonell, 2021).

Original: While discrete prompts have clear advantages, in addition to being human-readable and thus easily interpretable, we do not have efﬁcient and algorithmic ways of reconstructing them.
Revised: Most importantly, discrete prompts have the advantages of being human-readable and thus easily interpretable though we do not have efﬁcient and algorithmic ways of reconstructing them.

Original: For example, Shin et al.
Revised: For example, Shin et al.

Original: (2020)’s algorithm discovers discrete prompts, yet the results are not human readable.
Revised: (2020)’s algorithm discovers discrete prompts, yet the results are not human readable.

Original: Prior work also ﬁnds that model performance is highly sensitive to small changes in wordings (Mishra et al., 2021a) and that optimization over the discrete prompt space is nontrivial and often highly unstable.
Revised: Prior work also ﬁnds that model performance is highly sensitive to small changes in wordings (Mishra et al., 2022a) and that optimization over the discrete prompt space is non-trivial and often highly unstable.

Original: Our ﬁndings here about the disconnect between continuous prompts and their discrete interpretation provides another perspective on the difﬁculty of discovering discrete prompts via continuous optimization algorithms that (directly or indirectly) leverage the continuous space (more discussion in §6).
Revised: Our ﬁndings here about the disconnect between continuous prompts and their discrete interpretation provides another perspective on the difﬁculty of discovering discrete prompts via continuous optimization algorithms that (directly or indirectly) leverage the continuous space (more discussion in §6).

Original:  Prompt Waywardness 
Revised:  Prompt Waywardness 3. Preliminaries: Setup and Terminology

Original: 3. Preliminaries: Setup and Terminology

Original: We begin with some notation and the setup of our study, starting with the space of discrete and continuous prompts (Fig.2).
Revised: We begin with some notation and the setup of our study, starting with the space of discrete and continuous prompts (Fig.2).

Original: Let p d ∈ { 0 , 1 } L × V denote a discrete prompt represented as an L -length sequence of one-hot vectors over a lexicon of size V (corners of a hyper-cube).
Revised: Let p d ∈ { 0 , 1 } L × V denote a discrete prompt represented as an L -length sequence of one-hot vectors over a lexicon of size V (corners of a hyper-cube).

Original: Similarly, let p c ∈ R L ×
Revised: Similarly, let p c ∈ R L ×

Original: d denote a continuous prompt, represented as a Llength sequence of d -dimensional real vectors.
Revised: d denote a continuous prompt, represented as a Llength sequence of d -dimensional real vectors.

Original: Projection operators.
Revised: Projection operators.

Original: We deﬁne operators that project these two spaces to one another.
Revised: We deﬁne operators that project these two spaces to one another.

Original: Deﬁne the c -projection as one that maps discrete inputs to a continuous space by multiplying with a ﬁxed (often pre-trained) embedding matrix 2 E ∈ R V
Revised: Deﬁne the c -projection as one that maps discrete inputs to a continuous space by multiplying with a ﬁxed (often pre-trained) embedding matrix [Equation]

Original: × d :
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: The d -projection maps the continuous inputs to nearest neighbor discrete elements, where for each position l ( 1 ≤ l ≤ L ), one of the possible (and perhaps most straightforward) methods for interpreting a continuous prompt is deﬁned as a projection  onto nearest neighbor representations (Mikolov et al., 2013; Hashimoto et al., 2016):
Revised: The d -projection maps the continuous inputs to nearest neighbor discrete elements, where for each position l ( 1 ≤ l ≤ L ), one of the possible (and perhaps most straightforward) methods for interpreting a continuous prompt is deﬁned as a projection [Figure] onto nearest neighbor representations (Mikolov et al

Original: [Equation]
Revised: [Equation]

Revised: , 2013; Hashimoto et al., 2016):

Original: [Figure]
Revised: [Equation]

Original: where δ l is a one-hot vector corresponding to theword with the closest (highest dot product) embedding to the l -th position of continuous prompt p c .
Revised: where δ l is a one-hot vector corresponding to the word with the closest (highest dot product) embedding to the l -th position of continuous prompt p c .

Original: These projections are used in the ﬁrst and lastlayer of virtually all modern LMs, such as GPT2.
Revised: These projections are used in the ﬁrst and last layer of virtually all modern LMs, such as GPT2.

Original: Solving tasks with continuous prompts.
Revised: Solving tasks with continuous prompts.

Original: Consider any machine learning model M (typically apre-trained model) that takes textual input x andproduces output y .
Revised: Consider any machine learning model M (typically a pre-trained model) that takes textual input x and produces output y .

Original: Normally, the parameters of
Revised: Normally, the parameters of

Original: M are learned so as to optimize behavior on atask with a dataset D
Revised: M are learned so as to optimize behavior on a task with a dataset D

Original: = { ( x, y ) } of input/outputpairs.
Revised: = { ( x, y ) } of input/output pairs.

Original: In prompt tuning (Lester et al., 2021), onefreezes the parameters of M and instead optimizesfor a prompt p that, when fed in conjunction withx , makes M produce the desired output y .
Revised: In prompt tuning (Lester et al., 2021), one freezes the parameters of M and instead optimizes for a prompt p that, when fed in conjunction with x , makes M produce the desired output y .

Original: Thus,p represents the only learnable parameters in thismethod.
Revised: Thus, p represents the only learnable parameters in this method.

Original: When p is a discrete prompt with k tokens, it can be simply concatenated with x , denotedp + x .
Revised: When p is a discrete prompt with k tokens, it can be simply concatenated with x , denoted p + x .

Original: In our study, p will be a continuous prompt(of length equal to the embedding of k tokens).
Revised: In our study, p will be a continuous prompt (of length equal to the embedding of k tokens).

Original: Wewill concatenate it with the embedding of the inputx .
Revised: We will concatenate it with the embedding of the input x .

Original: For simplicity and with some abuse of notation, we use p + x to denote concatenation in thiscontinuous case as well.
Revised: For simplicity and with some abuse of notation, we use p + x to denote concatenation in this continuous case as well.

Original: One can quantify the amount of loss incurredwhen using a continuous prompt p as follows: 
Revised: One can quantify the amount of loss incurred when using a continuous prompt p as follows: (cid:96) ( p ; D ) =

Original: [Equation]
Revised: E x,y ∼D [ loss ( M ( p + x ) , y )] , (3)

Original: Minimizing this loss function (empirical risk minimization) over p recovers a minimum risk continuous prompt for this dataset:
Revised: Minimizing this loss function (empirical risk minimization) over p recovers a minimum risk continuous prompt for this dataset:

Original: [Equation]
Revised: [Equation]

Original: Given this prompt, its generalization to the test datacan be measured in terms of the loss incurred onthe test set: (cid:96) ( p ∗ c ; D test ) .
Revised: Given this prompt, its generalization to the test data can be measured in terms of the loss incurred on the test set: (cid:96) ( p ∗ c ; D test ) .

Original: 3. The Waywardness Hypothesis
Revised: 3. The Waywardness Hypothesis

Original: How should one interpret the resultant continuous prompt ˜ p c ?
Revised: How should one interpret the resultant continuous prompt ˜ p c ?

Original: Empirically, one can easily verify that such continuous prompts are not unique (e.g., random initializations lead to different outcomes).
Revised: Empirically, one can easily verify that such continuous prompts are not unique (e.g., random initializations lead to different outcomes).

Original: Additionally, the resultant prompts get projected to seemingly irrelevant discrete elements.
Revised: Additionally, the resultant prompts get projected to seemingly irrelevant discrete elements.

Original: Taking this to an extreme, we hypothesize that next to the continuous projection c -proj ( p d ) of any discrete prompt p d , there exists a variety of continuous prompts p c that trigger responses from model M that are orthogonal to the intentions described by the discrete prompt p d .
Revised: Taking this to an extreme, we hypothesize that next to the continuous projection c -proj ( p d ) of any discrete prompt p d , there exists a variety of continuous prompts p c that trigger responses from model M that are orthogonal to the intentions described by the discrete prompt p d .

Original: We formalize this idea as the following hypothesis, where L ∈ N is the length of the discrete target prompt, M is a promptbased model, and D is a dataset for a desired task:
Revised: We formalize this idea as the following hypothesis, where L ∈ N is the length of the discrete target prompt, M is a promptbased model, and D is a dataset for a desired task:

Original: Hypothesis 1 (Prompt Waywardness) For all
Revised: Hypothesis 1 (Prompt Waywardness) For all

Original: L, M, D , there is a small ∆ such that for any discrete target prompt p d with length L , there exists a continuous prompt ˜ p c ∈ R L × d such that:
Revised: L, M, D , there is a small ∆ such that for any discrete target prompt p d with length L , there exists a continuous prompt ˜ p c ∈ R L × d such that:

Original: 1. (cid:12)(cid:12) (cid:96) (˜ p c ; D test ) − (cid:96) ( p ∗ c ; D test ) (cid:12)(cid:12) < ∆ , yet 2. d -proj (˜ p c ) = p d .
Revised: [Equation]

Original: In other words, ˜ p c is nearly as effective at making
Revised: In other words, ˜ p c is nearly as effective at making

Original: M solve the task as the optimal continuous prompt (Eq.4), and yet it projects to p d .
Revised: M solve the task as the optimal continuous prompt (Eq.4), and yet it projects to p d .

Original: In this statement, ∆ (prompt performance gap relative to the optimal prompt p ∗ c ) is a function of the prompt length L , the model M (e.g., its embedding size and depth when
Revised: In this statement, ∆ (prompt performance gap relative to the optimal prompt p ∗ c ) is a function of the prompt length L , the model M (e.g., its embedding size and depth when

Original: M is transformer based), and inherent properties of the target dataset D .
Revised: M is transformer based), and inherent properties of the target dataset D .

Original: The analysis in §4.3 will provide an empirical estimate of this gap ˆ∆ as a function of various parameters like model size and prompt length.
Revised: The analysis in §4.3 will provide an empirical estimate of this gap ˆ∆ as a function of various parameters like model size and prompt length.

Original: It is worth emphasizing that the hypothesis is stated for any task and any set of discrete prompts, even if they are irrelevant or contradictory .
Revised: It is worth emphasizing that the hypothesis is stated for any task and any set of discrete prompts, even if they are irrelevant or contradictory .

Original: 3. Finding Wayward Prompts
Revised: 3. Finding Wayward Prompts

Original: While the above hypothesis promises the existence of ˜ p c , it does not say how to discover them.
Revised: While the above hypothesis promises the existence of ˜ p c , it does not say how to discover them.

Original: We now discuss a practical method for their discovery.
Revised: We now discuss a practical method for their discovery.

Original: We learn a continuous prompt p c using a modiﬁcation of the prompt tuning objective of Lester et al.
Revised: We learn a continuous prompt p c using a modiﬁcation of the prompt tuning objective of Lester et al.

Original: Our modiﬁcation jointly minimizes the standard downstream task cross-entropy loss (cid:96) ( . ) for the task (Eq.3) and a distance measuredist ( . ) between p c and the discrete target prompt [Equation] [Equation]
Revised: Our modiﬁcation jointly minimizes the standard downstream task cross-entropy loss (cid:96) ( . ) for the task (Eq.3) and a distance measure dist ( . ) between p c and the discrete target prompt p d ∈ { 0 , 1 } L × V :

Original: [Equation]
Revised: [Equation]

Original: where p c is the only learnable parameter, and γ isa hyperparamter.
Revised: where p c is the only learnable parameter, and γ is a hyperparameter.

Original: When γ = 0 , the modiﬁed objective is reducedto the standard objective (Eq.4), (cid:96) (cid:48) ( . )
Revised: When γ = 0 , the modiﬁed objective is reduced to the standard objective (Eq.4), (cid:96) (cid:48) ( . )

Original: = (cid:96) ( . ) .
Revised: = (cid:96) ( . ) .

Original: Werefer to this case and its resulting prompt p ∗ c asthe ‘unconstrained’ setting.
Revised: We refer to this case and its resulting prompt p ∗ c as the ‘unconstrained’ setting.

Original: A large value of γwill make p c even closer (possibly identical) toc -proj ( p d ) but lead to poor accuracy on a targetdataset.
Revised: A large value of γ will make p c even closer (possibly identical) to c -proj ( p d ) but lead to poor accuracy on a target dataset.

Original: Most of the experiments below are conducted via a range of γ values to better understandthe trade off between the two terms in the objectivefunction.
Revised: Most of the experiments below are conducted via a range of γ values to better understand the trade off between the two terms in the objective function.

Original: In practice, we ﬁnd γ = 0 . 01 to give areasonable trade-off regardless of the target datasetand the choice of p d .
Revised: In practice, we ﬁnd γ = 0 . 01 to give a reasonable trade-off regardless of the target dataset and the choice of p d .

Original: There are at least two natural ways to deﬁne thedistance measure dist ( p c , p d ) between a continuous prompt p c and a discrete target prompt p d , byconverting one so that both are in the same space:[Equation]
Revised: There are at least two natural ways to deﬁne the distance measure dist ( p c , p d ) between a continuous prompt p c and a discrete target prompt p d , by converting one so that both are in the same space:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst of these places both p c and p d in the continuous space and computes the squared-L 2 norm, normalized by the prompt length.
Revised: The ﬁrst of these places both p c and p d in the continuous space and computes the squared-L 2 norm, normalized by the prompt length.

Original: This is used in ourtraining loss (Eq.5) implementation.
Revised: This is used in our training loss (Eq.5) implementation.

Original: The secondplaces both in discrete space (text) and computesthe standard word-level token overlap F1 score.
Revised: The second places both in discrete space (text) and computes the standard word-level token overlap F1 score.

Original: This is used during our evaluation.
Revised: This is used during our evaluation.

Original:  Empirical Support of Waywardness
Revised:  Empirical Support of Waywardness

Original: We empirically investigate the Prompt Waywardness hypothesis (§3.2) using our modiﬁcation(§3.3) of the prompt tuning method from Lesteret al.
Revised: We empirically investigate the Prompt Waywardness hypothesis (§3.2) using our modiﬁcation (§3.3) of the prompt tuning method from Lester et al.

Original: We show that given an arbitraryand irrelevant discrete prompt p d , it is possibleto learn a continuous prompt that is mapped to p dwhile retaining its accuracy on a given dataset.
Revised: We show that given an arbitrary and irrelevant discrete prompt p d , it is possible to learn a continuous prompt that is mapped to p d while retaining its accuracy on a given dataset.

Original: 4. Setup
Revised: 4. Setup

Original: Target tasks.
Revised: Target tasks.

Original: Following the setup of Min et al.(2021), we select a diverse set of 5 classiﬁcation datasets:
Revised: Following the setup of Min et al. (2022), we select a diverse set of 5 classiﬁcation datasets:

Original: [Equation] (Socher et al., 2013),
Revised: SST-2 (Socher et al., 2013),

Original: SST-5 (Socher et al., 2013), AGNews (Zhang et al., 2015), Subj (Pang and Lee, 2004) and
Revised: SST-5 (Socher et al., 2013), AGNews (Zhang et al., 2015), Subj (Pang and Lee, 2004) and

Original: TREC (Voorhees and Tice, 2000).
Revised: TREC (Voorhees and Tice, 2000).

Original: Statistics and the unconstrained accuracy of each dataset are provided in Table 1.
Revised: Statistics and the unconstrained accuracy of each dataset are provided in Table 1.

Original: [Table]
Revised: [Table]

Original: ) using GPT2 Large, as a reference point.

Original: Discrete Target Projections.
Revised: Discrete Target Projections.

Original: We compile two sets of discrete target prompts: (1) 32 target prompts for solving tasks from NaturalInstructions 5 dataset (Mishra et al., 2021b) that are distinct from and intentionally orthogonal to the end tasks considered here.
Revised: We compile two sets of discrete target prompts: (1) 32 target prompts for solving tasks from NaturalInstructions 6 dataset (Mishra et al., 2022b) that are distinct from and intentionally orthogonal to the end tasks considered here.

Original: These were chosen by excluding discrete target prompts that have high lexical overlap with other discrete prompts; this is because we found lexically similar prompts are often semantically similar even when written for different subtasks.
Revised: These were chosen by excluding discrete target prompts that have high lexical overlap with other discrete prompts; this is because we found lexically similar prompts are often semantically similar even when written for different subtasks.

Original: (2) 30 random sentences from
Revised: (2) 30 random sentences from

Original: PILE, 6 a large-scale, diverse text corpus used to pretrain GPT-J, the largest public causal language model (Wang and Komatsuzaki, 2021).
Revised: PILE, 7 a large-scale, diverse text corpus used to pretrain GPT-J, the largest public causal language model (Wang and Komatsuzaki, 2021).

Original: The sampled sentences were drawn from a Poisson distribution with λ = 14 , which makes the average length of the sentence to be consistent to those in NaturalInstructions.
Revised: The sampled sentences were drawn from a Poisson distribution with λ = 14 , which makes the average length of the sentence to be consistent to those in NaturalInstructions.

Original: These sentences are selected to have little or no token overlap with the true deﬁnition of the target tasks.
Revised: These sentences are selected to have little or no token overlap with the true deﬁnition of the target tasks.

Original: See Table 3 for a few examples.
Revised: See Table 3 for a few examples.

Revised: Data

Revised: [Table]

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For all experiments, we report two metrics: (1) the task accuracy 7 as well as (2) prompt F1 , the word-level token overlap F score computed as in Eq.8, since it easy to interpret and is commonly used for evaluating the textual output of models (Rajpurkar et al., 2016).
Revised: For all experiments, we report two metrics: (1) the task accuracy 8 as well as (2) prompt F1 , the word-level token overlap F score computed as in Eq.8, since it easy to interpret and is commonly used for evaluating the textual output of models (Rajpurkar et al., 2016).

Original: Models.
Revised: Models.

Original: For evaluation, we use GPT2 (Radford et al., 2019) an auto-regressive LM which has extensively been used in many NLP applications.Un-
Revised: For evaluation, we use GPT2 (Radford et al., 2019) an auto-regressive LM which has extensively been used in many NLP applications.

Original: Data [Table] less otherwise speciﬁed, we use a ‘large’ variantconsisting of 774M parameters.
Revised: Unless otherwise speciﬁed, we use a ‘large’ variant consisting of 774M parameters.

Original: Implementation details.
Revised: Implementation details.

Original: We use a batch size of8, learning rate 0.01, and 2000 training steps.
Revised: We use a batch size of 8, learning rate 0.01, and 2000 training steps.

Original: Whenexperimenting with a discrete target prompt p d , weinitialize the search for continuous prompts (both˜ p c and p ∗ c ) using c -proj ( p d ) .
Revised: When experimenting with a discrete target prompt p d , we initialize the search for continuous prompts (both ˜ p c and p ∗ c ) using c -proj ( p d ) .

Original: For all experiments,report accuracy averaged over three random seeds.
Revised: For all experiments, report accuracy averaged over three random seeds.

Original: 4. Main Results
Revised: 4. Main Results

Original: For each of the 5 tasks T and for each of the 62discrete target prompts p d , we use the objectivein Eq.5 to ﬁnd a prompt ˜ p c such that it solves Twith a high accuracy while, at the same time, having a discrete projection that is close to p d .
Revised: For each of the 5 tasks T and for each of thediscrete target prompts p d , we use the objective in Eq.5 to ﬁnd a prompt ˜ p c such that it solves T with a high accuracy while, at the same time, having a discrete projection that is close to p d .

Original: Forcomparison, we also train unconstrained promptsp ∗ c ( γ = 0 . 0 ) which solve task T without any projection constraint.
Revised: For comparison, we also train unconstrained prompts p ∗ c ( γ = 0 . 0 ) which solve task T without any projection constraint.

Original: To ensure a fair comparisonbetween ˜ p c and p ∗ c , we ensure that they have thesame size L .
Revised: To ensure a fair comparison between ˜ p c and p ∗ c , we ensure that they have the same size L .

Original: In other words, for each ˜ p c (that hasthe same length as p d ), we train another p ∗ c with the same length.
Revised: In other words, for each ˜ p c (that has the same length as p d ), we train another p ∗ c with the same length.

Original: We denote the relative accuracy drop from p ∗ c to ˜ p c as ˆ∆ .
Revised: We denote the relative accuracy drop from p ∗ c to ˜ p c as ˆ∆ .

Original: Table 2 summarizes the results.
Revised: Table 2 summarizes the results.

Original: Across all datasets, we ﬁnd that it is possible to learn a continuous prompt p c whose discrete projection is very close to p d and mostly retains the task accuracy.
Revised: Across all datasets, we ﬁnd that it is possible to learn a continuous prompt p c whose discrete projection is very close to p d and mostly retains the task accuracy.

Original: There is a trade-off between the task accuracy and prompt F1, which can be controlled by the choice of γ (more extensive ablations in the forthcoming paragraphs (§4.3)).
Revised: There is a trade-off between the task accuracy and prompt F1, which can be controlled by the choice of γ (more extensive ablations in the forthcoming paragraphs (§4.3)).

Original: Overall, with γ = 0 .
Revised: Overall, with γ = 0 .

Original: 01 , it is possible to achieve ≥ 94% prompt F1 with under 2% relative drop in task accuracy.
Revised: 01 , it is possible to achieve ≥ 94% prompt F1 with under 2% relative drop in task accuracy.

Original: The only outlier is the TREC dataset where we achieved a prompt F1 score of 86% for a ˆ∆ = 2 .
Revised: The only outlier is the TREC dataset where we achieved a prompt F1 score of 86% for a ˆ∆ = 2 .

Original: 3% relative drop in accuracy.
Revised: 3% relative drop in accuracy.

Original: This might be due to the difﬁculty of learning effective prompts on TREC (also discussed by Min et al.
Revised: This might be due to the difﬁculty of learning effective prompts on TREC (also discussed by Min et al.

Original: Example prompts with varying values of prompt
Revised: Example prompts with varying values of prompt

Original: F1 scores are shown in Table 3.
Revised: F1 scores are shown in Table 3.

Original: A prompt F1 ≥ 94% generally indicates one word mismatch with almost no semantically meaningful difference.
Revised: A prompt F1 ≥ 94% generally indicates one word mismatch with almost no semantically meaningful difference.

Original: 4. Further Analysis
Revised: 4. Further Analysis

Original: Effect of Gamma.
Revised: Effect of Gamma

Original: Fig.
Revised: Fig.

Original: 3 shows the trade-off between task accuracy and the prompt F1 when varying γ from 0 to 0.03.
Revised: 3 shows the trade-off between task accuracy and the prompt F1 when varying γ from 0 to 0.03.

Original: As γ increases, the task accuracy goes down while the prompt F1 increases.
Revised: As γ increases, the task accuracy goes down while the prompt F1 increases.

Original: The drop in task accuracy is relatively minor—it is possible to learn a continuous prompt for which prompt F1 is near 1.00 and the accuracy drop relative to the unconstrained accuracy is less than 1%.
Revised: The drop in task accuracy is relatively minor—it is possible to learn a continuous prompt for which prompt F1 is near 1.00 and the accuracy drop relative to the unconstrained accuracy is less than 1%.

Original: Effect of Prompt Length ( L ).
Revised: Effect of Prompt Length ( L ).

Original: We randomly sample sentences from The PILE with a constraint that its length must be L (chosen from { 4 , 7 , 14 , 28 , 56 } ).
Revised: We randomly sample sentences from The PILE with a constraint that its length must be L (chosen from { 4 , 7 , 14 , 28 , 56 } ).

Original: The left and the middle parts of
Revised: The left and the middle parts of

Original: Fig.
Revised: Fig.

Original: 4 illustrate the results.
Revised: 4 illustrate the results.

Original: We ﬁnd that when L is very small (e.g., 4) it is relatively difﬁcult to learn a continuous prompt p c that is close to p d
Revised: We ﬁnd that when L is very small (e.g., 4) it is relatively difﬁcult to learn a continuous prompt p c that is close to p d

Original: (F1<60%) while retaining the task accuracy.
Revised: (F1<60%) while retaining the task accuracy.

Original: This is likely because the prompt being too short signiﬁcantly hurts the expressivity of the prompt.

Original: Nonetheless, when

Original: L is reasonably larger, e.g., 14 (the average length of in Natural Instructions) or longer, all cases lead to a continuous prompt with near 1.0 prompt F  and little accuracy drop.

Original: Effect of Model Size.

Original: We vary the size of the

Original: GPT2 models—small, medium, large, and XL— with 124M, 355M, 774M, and 1.5B parameters, respectively. Figure 5 (right) reports the result on SST-2. We ﬁnd that (1) across different sizes of the
Revised: This is likely be-

Original: LM, our ﬁndings in learning continuous prompts

Original: [Table]
Revised: [Table]

Original: int
Revised: int

Original: [Table]
Revised: [Table]

Original: return
Revised: return

Original: [Table]
Revised: [Table]

Original: std
Revised: std

Original: [Table]
Revised: [Table]

Original: min
Revised: min

Original: [Table]
Revised: [Table]

Original: int val
Revised: int val

Original: [Table]
Revised: [Table]

Original: int min
Revised: int min

Original: [Table]
Revised: [Table]

Original: val
Revised: val

Original: [Table]
Revised: [Table]

Original: int max
Revised: int max

Original: [Table]
Revised: [Table]

Original: val
Revised: val

Original: [Table]
Revised: [Table]

Original: return std
Revised: return std

Original: [Table]
Revised: [Table]

Original: max
Revised: max

Original: [Table]
Revised: [Table]

Original: min
Revised: min

Original: [Table]
Revised: [Table]

Original: val
Revised: val

Original: [Table]
Revised: [Table]

Original: std
Revised: std

Original: [Table]
Revised: [Table]

Original: min
Revised: min

Original: [Table]
Revised: [Table]

Original: max
Revised: max

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: cause the prompt being too short signiﬁcantly hurts the expressivity of the prompt.

Original: with the prompt F1 of  near 1.0  and littledrop inthe accuracy generally hold, and (2) in particular,the drop in accuracy is more negligible with larger
Revised: Nonetheless, when L is reasonably larger, e.g., 14 (the average length of in Natural Instructions) or longer, all cases lead to a continuous prompt with near 1.0 prompt F and little accuracy drop. Effect of Model Size. We vary the size of the

Revised: GPT2 models—small, medium, large, and XL— with 124M, 355M, 774M, and 1.5B parameters, respectively.

Revised: Figure 5 (right) reports the result on

Revised: SST-2.

Revised: We ﬁnd that (1) across different sizes of the

Revised: LM, our ﬁndings in learning continuous prompts with the prompt F1 of near 1.0 and little drop in the accuracy generally hold, and (2) in particular, the drop in accuracy is more negligible with larger

Original: LMs (0.2% with XL, 0.5–0.7% with medium andlarge, 1.2% with small).
Revised: LMs (0.2% with XL, 0.5–0.7% with medium and large, 1.2% with small).

Revised: Projection onto true task deﬁnitions.

Revised: In all our results so far in §4, the target projected text was

Original: Explaining Waywardness
Revised: [Table] orthogonal to the tasks being solved. One might naturally wonder whether there is any beneﬁt in projecting continuous prompts to the texts that truly describe the task being solved, i.e., a “true” prompt for the task. To this end, we manually authored

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: p̃_c p

Revised: [Figure]

Revised: c

Revised: [Figure]

Revised: “true” prompts for each of the tasks.

Revised: We then follow the exact same setup used earlier for Table 2 to ﬁne-tune continuous prompts ˜ p c for the task while projecting onto these true task deﬁnitions.

Revised: As before, we ﬁne-tune unconstrained prompts p ∗ c of the same length, without any projection requirement.

Revised: By design, ˜ p c can be no more effective at solving the task than the unconstrained prompt p ∗ c (barring suboptimal search issues), which is what we ﬁnd in practice.

Revised: For completeness, we report detailed results for “true” target prompts (analogous to Table 2) in Table 4.

Revised: More interestingly, as shown in Table 5, continuous prompts that project to “true” target prompts are no more effective at solving the task than continuous prompts that project to the 62 irrelevant target prompts considered earlier (Table 2).

Revised: Speciﬁcally, the average performance gap ∆ (relative to unconstrained prompts of the same length) is about the same ( ≈ 1 . 5% ) for continuous prompts that map to true task deﬁnitions compared to prompts that map to irrelevant text.

Revised: This further bolsters the waywardness hypothesis—continuous prompts don’t relate to the task being solved.

Revised: [Figure]

Revised: Large

Revised: XL p̃_c p*_c

Revised: We vary γ = { 0 . 01 , 0 .

Revised: 005 , 0 . 003 } and choose the one for which prompt F1 is larger than 0.98.

Revised: The relative accuracy drop (gap between the two trends) decreases as models become larger.

Revised: [Figure]

Revised: AGNews

Revised: [Figure]

Revised: TREC Avg ˆ∆ T 1.

Revised: ˆ∆ 0.

Revised: perfor irrelevant

Revised:  Explaining Waywardness

Original: Here we provide intuitions behind the factors thatenable Prompt Waywardness.
Revised: Here we provide intuitions behind the factors that enable Prompt Waywardness.

Original: The mapping between continuous and discretespaces is not one-to-one.
Revised: The mapping between continuous and discrete spaces is not one-to-one.

Original: While a discrete target prompt is mapped to exactly one continuousprompt (via its embedding, Eq.1; cf. Fig.2), thereverse is not true.
Revised: While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1; cf. Fig.2), the reverse is not true.

Original: In fact, except for a very smallfraction of unnatural or degenerate edge cases, 9for every target discrete prompt, there are inﬁnitelymany continuous prompts that project back to it (via Eq.2).
Revised: In fact, except for a very small fraction of unnatural or degenerate edge cases,for every target discrete prompt, there are inﬁnitely many continuous prompts that project back to it (via Eq.2).

Original: While simple counting-based arguments are insufﬁcient in continuous spaces, we formally prove (Appendix C) that this property holds for all nearest-neighbor projections under any metric distance, and broadly for all but a negligible (measure zero) portion of possible projection operators. 
Revised: While simple counting-based arguments are insufﬁcient in continuous spaces, we formally prove (Appendix B) that this property holds for all nearest-neighbor projections under any metric distance, and broadly for all but a negligible (measure zero) portion of possible projection operators. [Figure]

Revised: subspace of continuous prompts that have the same projections but solve different tasks.

Original: This intuitively suggests that there is a whole region of continuous prompts that corresponds to a ﬁxed discrete representation (Fig.6).
Revised: This intuitively suggests that there is a whole region of continuous prompts that corresponds to a ﬁxed discrete representation (Fig.6).

Original: The remaining question is, how is this region able to have a diverse set of prompts that can solve a variety of tasks?
Revised: The remaining question is, how is this region able to have a diverse set of prompts that can solve a variety of tasks?

Original: This is addressed next.
Revised: This is addressed next.

Original: Deep models give immense expressive power to earlier layers.
Revised: Deep models give immense expressive power to earlier layers.

Original: The deeper a network is, the more expressivity it has with respect to its inputs (Telgarsky, 2016; Raghu et al., 2017).
Revised: The deeper a network is, the more expressivity it has with respect to its inputs (Telgarsky, 2016; Raghu et al., 2017).

Original: Since continuous prompts reside just before the ﬁrst layer, they enjoy a lot of expressivity.
Revised: Since continuous prompts reside just before the ﬁrst layer, they enjoy a lot of expressivity.

Original: Therefore, no matter how narrow the regions corresponding to individual tokens are (Fig.6), they are extremely powerful in solvingA cc u r ac y ( % ) 80% 83% 85% 88% 90% 6 8 40p̃_c p*_c Prompt Length L (tokens) P r o m p t F 1 ( % ) 50% 60% 70% 80% 90% 100% 6 8 40 a variety of tasks.
Revised: Therefore, no matter how narrow the regions corresponding to individual tokens are (Fig.6), they are extremely powerful in solving a variety of tasks.

Original: Previously in §4.2 we provide anempirical analysis showing evidence that the effectof Waywardness is stronger in deeper models.
Revised: Previously in §4.2 we provide an empirical analysis showing evidence that the effect of Waywardness is stronger in deeper models.

Original:  Implications of Prompt Waywardness
Revised:  Implications of Prompt Waywardness

Original: We discuss the implications of these ﬁndings onseveral inter-related lines of research.
Revised: We discuss the implications of these ﬁndings on several inter-related lines of research.

Original: Note thatall the following statements are valid within theboundaries of the existing architectures.
Revised: Note that all the following statements are valid within the boundaries of the existing architectures.

Original: Movingbeyond these barriers likely requires major innovations in terms of LM architectures or how continuous prompts are optimized.
Revised: Moving beyond these barriers likely requires major innovations in terms of LM architectures or how continuous prompts are optimized.

Original: Faithful interpretation of continuous promptsis difﬁcult.
Revised: Faithful interpretation of continuous prompts is difﬁcult.

Original: Given the intuitions behind and empirical support for the Waywardness hypothesis(§5), faithful discrete interpretations of continuous prompts via common discrete projections (likenearest-neighbor projection) are unlikely to be robust based on current approaches.
Revised: Given the intuitions behind and empirical support for the Waywardness hypothesis (§5), faithful discrete interpretations of continuous prompts via common discrete projections (like nearest-neighbor projection) are unlikely to be robust based on current approaches.

Original: It is an openquestion whether there is a better way of interpreting continuous prompts with human language,or whether explaining and interpreting continuousprompts via human language is inherently impossible because they lie in completely different spaces.Future work may investigate more on this topic [Figure] u r ac y ( % ) 80% 85%
Revised: It is an open question whether there is a better way of interpreting continuous prompts with human language, or whether explaining and interpreting continuous prompts via human language is inherently impossible because they lie in completely different spaces.

Original: [Figure]

Original: [Figure]

Original: Large

Original: XL p̃_c p*_c

Original: We vary γ and

Original: F1 is larger than 0.98.

Original: relative decreases become larger [Figure] subspace of continuous prompts that have the same but solve different tasks. in order to improve the interpretability of promptbased language models.
Revised: Future work may investigate more on this topic in order to improve the interpretability of promptbased language models.

Original: Risk of interpreting continuous prompts: concealed adversarial attacks.
Revised: Risk of interpreting continuous prompts: concealed adversarial attacks.

Original: It is not difﬁcult to imagine a future where proprietary model development is driven by ﬁne-tuned continuous prompts.
Revised: It is not difﬁcult to imagine a future where proprietary model development is driven by ﬁne-tuned continuous prompts.

Original: In such a world, not addressing the challenges involved in discrete interpretation of continuous prompts can lead to harmful (and potentially, adversarial) consequences (Slack et al., 2020; Wallace et al., 2021), as discussed below. 
Revised: In such a world, not addressing the challenges involved in discrete interpretation of continuous prompts can lead to harmful (and potentially, adversarial) consequences (Slack et al., 2020; Wallace et al., 2021), as discussed below. [Figure]

Original: We consider the following scenario: a model designer comes up with a set of continuous prompts that solve a target task (e.g., ranking resumes according to each applicant’s qualiﬁcations and merits).
Revised: We consider the following scenario: a model designer comes up with a set of continuous prompts that solve a target task (e.g., ranking resumes according to each applicant’s qualiﬁcations and merits).

Original: Whether intentionally or not, such prompts may maliciously target, for example, a minority group.
Revised: Whether intentionally or not, such prompts may maliciously target, for example, a minority group.

Original: To assure their customers, the model designer uses the projection of the prompt that ex- [Figure] presses a benign deﬁnition for the task, which doesnot reveal the true nature of the egregious behavior.
Revised: To assure their customers, the model designer uses the projection of the prompt that expresses a benign deﬁnition for the task, which does not reveal the true nature of the egregious behavior.

Original: The customers might even evaluate the prompt ona few instances but not notice this harmful behavior, e.g., when it effects a minority group not inthe evaluation set.
Revised: The customers might even evaluate the prompt on a few instances but not notice this harmful behavior, e.g., when it effects a minority group not in the evaluation set.

Original: In a way, the benign discreteprojections may provide a false sense of security.
Revised: In a way, the benign discrete projections may provide a false sense of security.

Original: Optimizing discrete prompts through continuous prompts can be degenerate.
Revised: Optimizing discrete prompts through continuous prompts can be degenerate.

Original: Manuallywritten discrete prompts have many nice properties (Schick and Schütze, 2021;
Revised: Manuallywritten discrete prompts have many nice properties (Schick and Schütze, 2021;

Original: Mishra et al.,2021a), yet we do not have an efﬁcient algorithmicway of ﬁnding them.
Revised: Mishra et al., 2022a), yet we do not have an efﬁcient algorithmic way of ﬁnding them.

Original: One way to operationalize thisis to formulate differentiable objective functionsvia LMs like GPT (Radford et al., 2019).
Revised: One way to operationalize this is to formulate differentiable objective functions via LMs like GPT (Radford et al., 2019).

Original: Considerthe following problem which is deﬁned in the spaceof continuous embeddings p c ∈ R d :
Revised: Consider the following problem which is deﬁned in the space of continuous embeddings p c ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: This a joint optimization towards a utility objective(the extent to which it can solve dataset D ) anda human readability objective.
Revised: This a joint optimization towards a utility objective (the extent to which it can solve dataset D ) and a human readability objective.

Original: According to the
Revised: According to the

Original: Waywardness hypothesis, there are p c ’s that assignhigh mass to the utility term while also mapping tohuman interpretable text that is irrelevant (or evencontradictory) to the task solved by the prompt –hence, degenerate solutions.
Revised: Waywardness hypothesis, there are p c ’s that assign high mass to the utility term while also mapping to human interpretable text that is irrelevant (or even contradictory) to the task solved by the prompt – hence, degenerate solutions.

Original: The same challenge holds if this optimizationobjective, instead of continuous prompts, is reformulated in terms of word probabilities (e.g., similarto Kumar et al.
Revised: The same challenge holds if this optimization objective, instead of continuous prompts, is reformulated in terms of word probabilities (e.g., similar to Kumar et al.

Original: (2021, Sec 2.2)).
Revised: (2021, Sec 2.2)).

Original: This is the case,since searching in the space of word probabilitiesis analogous to a search in embedding spaces.
Revised: This is the case, since searching in the space of word probabilities is analogous to a search in embedding spaces.

Original: In summary, Waywardness presents a challenge for searching effective discrete prompts via continuous optimization.
Revised: In summary, Waywardness presents a challenge for searching effective discrete prompts via continuous optimization.

Original: The recent works have used additional signals such as domain-speciﬁc constraints (Qin et al., 2020; Khot et al., 2021) to alleviate these challenges.
Revised: The recent works have used additional signals such as domain-speciﬁc constraints (Qin et al., 2020; Khot et al., 2021; Qin et al., 2022) to alleviate these challenges.

Original: We hope to see more design innovationsfor further progress in this direction.
Revised: We hope to see more design innovations in this direction.

Original: Continuous prompt tuning does not necessitate task-speciﬁc initialization.
Revised: Gradients alone are insufﬁcient to reverse engineer a model. Suppose we are given a ﬁxed (ﬁne-tuned or otherwise) model M (e.g., an open question-answering model) and an expected output y from this model (e.g., y = “Joe Biden” ). Can we use gradients with respect to an LM’s output to ﬁnd a semantically meaningful input that makes a frozen model M generate a particular output? Our ﬁndings and the earlier argument about continuous differentiable optimization suggests this may not be feasible with current methods.

Revised: To see the correspondence to Prompt Waywardness, we can replace D in Eq.9 with the desired outcome y and run the optimization over word distributions (cf.

Revised: Footnote 11).

Revised: While gradients can guide towards some input that makes M produce y , their interpretation is likely unfaithful to the task being solved by M .

Revised: In the context of the above example ( M being a QA system), gradients might lead to inputs maximize the probability assigned to “Joe

Revised: Biden” , although this input will likely be neither ﬂuent nor semantically descriptive of “Joe Biden”.

Revised: Nevertheless, as noted earlier, gradients are still useful when they are applied using domain-speciﬁc constraints.

Revised: For example, one can ﬁnd local (wordlevel) perturbations that lead to a certain adversarial outcome, if the perturbations are restricted to welldeﬁned semantic categories (e.g., “blue” can be perturbed to any other color name) (Sha, 2020;

Revised: Guo et al., 2021; Yuan et al., 2021).

Revised: Continuous prompt tuning does not necessitate task-speciﬁc initialization.

Original: Recent works on continuous prompt-tuning have shown the effectiveness of initialization from embeddings of random common words (Lester et al., 2021;
Revised: Recent works on continuous prompt-tuning have shown the effectiveness of initialization from embeddings of random common words (Lester et al., 2021;

Original: Min et al., 2021), despite these words being irrelevant to the task solved by these prompts.
Revised: Min et al., 2022), despite these words being irrelevant to the task solved by these prompts.

Original: This, however, makes sense given the observations made in this work regarding the existence of effective prompts around word embedding subspaces.
Revised: This, however, makes sense given the observations made in this work regarding the existence of effective prompts around word embeddings.

Original:  Conclusion
Revised:  Conclusion

Original: The prompting literature has seen many parallel developments around continuous and discrete prompts, as efﬁcient alternatives to ﬁne-tuning models with tens of millions of parameters.
Revised: The prompting literature has seen many parallel developments around continuous and discrete prompts, as efﬁcient alternatives to ﬁne-tuning models with tens of millions of parameters.

Original: Our work introduced the Prompt Waywardness hypothesis, which expresses a surprising disconnect between continuous and discrete prompts: given a downstream task, for any discrete target prompt p d , there exists a continuous prompt that projects to p d while achieving strong performance on the task.
Revised: Our work introduced the Prompt Waywardness hypothesis, which expresses a surprising disconnect between continuous and discrete prompts: given a downstream task, for any discrete target prompt p d , there exists a continuous prompt that projects to p d while achieving strong performance on the task.

Original: We provided empirical evidence for this hypothesis, studied various parameters around it, and ended with several implications of this hypothesis.
Revised: We provided empirical evidence for this hypothesis, studied various parameters around it, and ended with several implications of this hypothesis.

Original: While our experiments are done on the GPT family, we expect our ﬁndings to apply to a broader set of architectures that, in one way or another, use similar mechanisms for mapping discrete elements to continuous representations and vice versa.
Revised: While our experiments are done on the GPT family, we expect our ﬁndings to apply to a broader set of architectures that, in one way or another, use similar mechanisms for mapping discrete elements to continuous representations and vice versa.

Original: Similarly, while our projection to the discrete space (Eq.2) is a popular operator in the ﬁeld (cf.
Revised: Similarly, while our projection to the discrete space (Eq.2) is a popular operator in the ﬁeld (cf.

Original: Footnote 1), the intuition explained in Propositions and 2 of the Appendix suggests similar behavior for a broad class of projection operators.
Revised: Footnote 1), the intuition explained in Propositionsand 2 of the Appendix suggests similar behavior for a broad class of projection operators.

Original:  future progress on algorithmic methods for the discovery of human readable prompts that are faithful to the task they solve.
Revised: Prompt Waywardness identiﬁes challenges for future progress on algorithmic methods for the discovery of human readable prompts that are faithful to the task they solve.

Original: We hope the observations made in this work motivate architectural innovations that overcome such challenges and guide future steps in the prompting literature.
Revised: We hope the observations made in this work motivate architectural innovations that overcome such challenges and guide future steps in the prompting literature.

Revised: Acknowledgment

Revised: The authors are thankful to Lisa Li, Nicholas

Revised: Lourie, and Vered Shwartz for helpful discussions, and the Beaker team at AI2 for their support with the experiments.

Revised: This work was supported in part by DARPA MCS program through NIWC Paciﬁc (N66001-19-2-4031), DARPA SemaFor program, and Google Cloud Compute.

Original: Supplementary Material 
Revised: Supplementary Material A Additional Experimental Details

Original: A Additional Experimental Details

Original: Here we include several experimental details (§4)that did not ﬁt in the main text.
Revised: Here we include several experimental details (§4) that did not ﬁt in the main text.

Original: For the experimentswe used A100 GPUs with 40G memory.
Revised: For the experiments we used A100 GPUs with 40G memory.

Original: In terms ofthe time GPU time of the experiments, each roundof training and inference for each seed took aboutaround 6 min.
Revised: In terms of the time GPU time of the experiments, each round of training and inference for each seed took about around 6 min.

Original: Therefore, the total GPU hours forour main experiment (Table 2) adds up to 93 hours(6 mins × 3 seeds × 5 datasets × 62 prompts =5580 mins).
Revised: Therefore, the total GPU hours for our main experiment (Table 2) adds up to 93 hours (6 mins × 3 seeds × 5 datasets × 62 prompts = 5580 mins).

Original: B Experiment: Projection onto TrueTask Deﬁnitions

Original: In all our results so far in §4, the target projectedtext was orthogonal to the tasks being solved.

Original: Onemight naturally wonder whether there is any beneﬁtin projecting continuous prompts to the texts thattruly describe the task being solved, i.e., a “true”prompt for the task.

Original: To this end, we manually authored 5 “true” prompts for each of the tasks.

Original: Wethen follow the exact same setup used earlier for

Original: Table 2 to ﬁne-tune continuous prompts ˜ p c for thetask while projecting onto these true task deﬁnitions.

Original: As before, we also ﬁne-tune unconstrainedprompts p ∗ c of the same length, without any projection requirement.

Original: By design, ˜ p c can be no more effective at solvingthe task than the unconstrained prompt p ∗ c (barringsuboptimal search issues), which is what we ﬁndin practice.

Original: For completeness, we report detailedresults for “true” target prompts (analogous to Table 2) in Table 4.

Original: More interestingly, as shown in Table 5, continuous prompts that project to “true” targetprompts are no more effective at solving thetask than continuous prompts that project to the62 irrelevant target prompts considered earlier (Table 2).

Original: Speciﬁcally, the average performance gap∆ (relative to unconstrained prompts of the samelength) is about the same (around 1.5) for continuous prompts that map to true task deﬁnitionscompared to prompts that map to irrelevant text.

Original: This observation further bolsters the waywardnesshypothesis—continuous prompts don’t relate to thetask being solved.

Original: [Table]

Original: [Table]

Original: erage performance gaps are about the same (around 1.5) for true and irrelevant target prompts —further evidence that continuous prompts don’t relate to the task being solved.

Original: C The mapping between continuous and 
Revised: B The mapping between continuous and discrete space is not one-to-one

Original: discrete space is not one-to-one

Original: As argued in §5, the mapping between the space of discrete input and that of word embeddings (Fig.2) is not a bijection.
Revised: As argued in §5, the mapping between the space of discrete input and that of word embeddings (Fig.2) is not a bijection.

Original: While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1), the reverse is not true: except for some unnatural or rare cases (as formalized in the following propositions) there are inﬁnitely many continuous prompts that project back to a ﬁxed discrete target prompt (via Eq.2).
Revised: While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1), the reverse is not true: except for some unnatural or rare cases (as formalized in the following propositions) there are inﬁnitely many continuous prompts that project back to a ﬁxed discrete target prompt (via Eq.2).

Original: Nearest-neighbor projections are arguably natural, computationally efﬁcient, and useful in practice.
Revised: Nearest-neighbor projections are arguably natural, computationally efﬁcient, and useful in practice.

Original: Although we have considered them in the
Revised: Although we have considered them in the

Original: Euclidean space so far, they can be deﬁned for an arbitrary distance metric 11 m on R d .
Revised: Euclidean space so far, they can be deﬁned for an arbitrary distance metric 12 m on R d .

Original: As before, consider an embedding of a lexicon of size
Revised: As before, consider an embedding of a lexicon of size

Original: V into R d and the corresponding one-hot vectors in { 0 , 1 } V .
Revised: V into R d and the corresponding one-hot vectors in { 0 , 1 } V .

Original: We call d -proj a nearest-neighbor projection operator w.r.t.
Revised: We call d -proj a nearest-neighbor projection operator w.r.t.

Original: m if it maps each x ∈ R d to the one-hot vector in { 0 , 1 } V that corresponds to the lexicon item whose embedding is closest to xunder metric m (breaking ties arbitrarily).
Revised: m if it maps each x ∈ R d to the one-hot vector in { 0 , 1 } V that corresponds to the lexicon item whose embedding is closest to x under metric m (breaking ties arbitrarily).

Original: Proposition 1 Every nearest-neighbor projection operator, under any metric, maps inﬁnitely many elements of R d , forming one or more continuous subspaces, to every one-hot vector in { 0 , 1 } V .
Revised: Proposition 1 Every nearest-neighbor projection operator, under any metric, maps inﬁnitely many elements of R d , forming one or more continuous subspaces, to every one-hot vector in { 0 , 1 } V .

Original: A proof is included in Appendix C.1.
Revised: A proof is included in Appendix B.1.

Original: In effect,the projection operators induce a clustering of thespace of continuous prompts R d × L into regionsthat have the same discrete projection (Fig.6).
Revised: In effect, the projection operators induce a clustering of the space of continuous prompts R d × L into regions that have the same discrete projection (Fig.6).

Original: The inﬁnite-to-one mapping aspect is not limitedto the class of nearest-neighbor projection operators.
Revised: The inﬁnite-to-one mapping aspect is not limited to the class of nearest-neighbor projection operators.

Original: It is rather an inherent property of the interaction between continuous and discrete spaces, andholds for a broader family consisting of all but anegligible portion of possible projection operators:
Revised: It is rather an inherent property of the interaction between continuous and discrete spaces, and holds for a broader family consisting of all but a negligible portion of possible projection operators:

Original: Proposition 2 Let D denote the space of all projection operators that map R d to one-hot vectors in { 0 , 1 } V .
Revised: Proposition 2 Let D denote the space of all projection operators that map R d to one-hot vectors in { 0 , 1 } V .

Original: Let d -proj be a random projection drawn uniformly from D .
Revised: Let d -proj be a random projection drawn uniformly from D .

Original: Then, with probability 1, d -proj maps inﬁnite elements of R d to every one-hot vector in { 0 , 1 } V .
Revised: Then, with probability 1, d -proj maps inﬁnite elements of R d to every one-hot vector in { 0 , 1 } V .

Original: C. Proofs
Revised: B. Proofs

Original: Proof of Prop. 1:
Revised: Proof of Prop. 1:

Original: Let c i ∈
Revised: Let c i ∈

Original: R d for i ∈{ 1 , . . .
Revised: R d for i ∈ { 1 , . . .

Original: , V } be ﬁxed vectors (denoting the embedding of words in a lexicon of size V ).
Revised: , V } be ﬁxed vectors (denoting the embedding of words in a lexicon of size V ).

Original: Lete i ∈ { 0 , 1 } V denote the one-hot vector with 1 inthe i -th position and 0 elsewhere.
Revised: Let e i ∈ { 0 , 1 } V denote the one-hot vector with 1 in the i -th position and 0 elsewhere.

Original: Since d -proj is anearest-neighbor projection operator w.r.t.
Revised: Since d -proj is a nearest-neighbor projection operator w.r.t.

Original: m , bydeﬁnition it maps x ∈ R d to e i whenever x is closest to c
Revised: m , by deﬁnition it maps x ∈ R d to e i whenever x is closest to c

Original: i , i.e., i = arg min
Revised: i , i.e., i = arg min

Original: j m ( x, c j ) (breaking tiesarbitrarily).
Revised: j m ( x, c j ) (breaking ties arbitrarily).

Original: Let S i ⊆
Revised: Let S i ⊆

Original: R d denote the pre-image of e i , i.e.,the elements that the nearest-neighbor projectiond -proj maps to the i -th one-hot vector.
Revised: R d denote the pre-image of e i , i.e., the elements that the nearest-neighbor projection d -proj maps to the i -th one-hot vector.

Original: By deﬁnition, c i ∈ S i .
Revised: By deﬁnition, c i ∈ S i .

Original: Let d (cid:48) =
Revised: Let d (cid:48) =

Original: min j m ( c i , c j ) > 0 denotethe distance of c i to the nearest c j w.r.t.
Revised: min j m ( c i , c j ) > 0 denote the distance of c i to the nearest c j w.r.t.

Original: the metricm .
Revised: the metric m .

Original: Consider the subspace C i
Revised: Consider the subspace C i

Original: = { x | m ( x, c i ) <
Revised: = { x | m ( x, c i ) <

Original: d (cid:48) / 2 } .
Revised: d (cid:48) / 2 } .

Original: By design, we have C i ⊆ S i .
Revised: By design, we have C i ⊆ S i .

Original: Further, moving x by some small distance (cid:15) (w.r.t.
Revised: Further, moving x by some small distance (cid:15) (w.r.t.

Original: m ) to anotherpoint x (cid:48) changes its distance to c i only by at most(cid:15) (by the triangle inequality property of m ).
Revised: m ) to another point x (cid:48) changes its distance to c i only by at most (cid:15) (by the triangle inequality property of m ).

Original: Thisimplies that if (cid:15) is chosen to be small enough suchthat m ( x, c i ) + (cid:15) <
Revised: This implies that if (cid:15) is chosen to be small enough such that m ( x, c i ) + (cid:15) <

Original: d (cid:48) / 2 , then x (cid:48) must also be in
Revised: d (cid:48) / 2 , then x (cid:48) must also be in

Original: C i .
Revised: C i .

Original: In other words, if x ∈ C i , then, for a smallenough (cid:15) , the entire (cid:15) -neighborhood of x is also in
Revised: In other words, if x ∈ C i , then, for a small enough (cid:15) , the entire (cid:15) -neighborhood of x is also in

Original: C i .
Revised: C i .

Original: It follows that C i is an open subset of R d andthus contains inﬁnitely many elements forming acontinuous subspace.
Revised: It follows that C i is an open subset of R d and thus contains inﬁnitely many elements forming a continuous subspace.

Original: Hence S i , which contains
Revised: Hence S i , which contains

Original: C i ,e.g. also has inﬁnite elements in one or more continuous subspaces.
Revised: C i ,e.g. also has inﬁnite elements in one or more continuous subspaces.

Original: (cid:3)
Revised: [Equation]

Original: Proof of Prop. 2:
Revised: Proof of Prop. 2:

Original: For simplicity, assume V = 2 .
Revised: For simplicity, assume V = 2 .

Original: A projection operator d -proj ∈ D can then be fully characterized by the subset S ⊆ R d that it maps to any one arbitrarily chosen one-hot vector.
Revised: A projection operator d -proj ∈ D can then be fully characterized by the subset S ⊆ R d that it maps to any one arbitrarily chosen one-hot vector.

Original: Choosing d -proj uniformly at random from D thus amounts to choosing the subset S uniformly at random from R d .
Revised: Choosing d -proj uniformly at random from D thus amounts to choosing the subset S uniformly at random from R d .

Original: We show that the probability of choosing an S such that | S | is ﬁnite, is 0.
Revised: We show that the probability of choosing an S such that | S | is ﬁnite, is 0.

Original: (The same argument applies to | R \ S | being ﬁnite.)
Revised: (The same argument applies to | R \ S | being ﬁnite.)

Original: To see this, let S i denote the set of all (ﬁnite) subsets of R d that have size exactly i .
Revised: To see this, let S i denote the set of all (ﬁnite) subsets of R d that have size exactly i .

Original: First, observe that the probability of choosing an S that lies in S 0 ∪ S 1 (i.e., a subset of R d that has at most 1 element) is 0; this is a degenerate case in the underlying continuous probability space.
Revised: First, observe that the probability of choosing an S that lies in S 0 ∪ S 1 (i.e., a subset of R d that has at most 1 element) is 0; this is a degenerate case in the underlying continuous probability space.

Original: Second, for any i ≥ 2 , S i has the same “size” (in the measure theoretic sense) as S 1 , because one can construct an injective map from either one to the other—which follows from the fact that they both have the same cardinality as the set R .
Revised: Second, for any i ≥ 2 , S i has the same “size” (in the measure theoretic sense) as S 1 , because one can construct an injective map from either one to the other—which follows from the fact that they both have the same cardinality as the set R .

Original: Lastly, the space S of all ﬁnite subsets of R d is the countable union ∪ i S i of disjoint sets.
Revised: Lastly, the space S of all ﬁnite subsets of R d is the countable union ∪ i S i of disjoint sets.

Original: Therefore,
Revised: Therefore,

Original: [Equation]

Original: D Implications of Prompt Waywardness:

Original: continued

Original: Here we mention other implications (§6) that did not ﬁt in our page limit.

Original: Gradients alone are insufﬁcient to reverse engineer a model.

Original: Suppose we are given a ﬁxed (ﬁne-tuned or otherwise) model M (e.g., an open question-answering model) and an expected output y from this model (e.g., y = “Joe Biden” ).

Original: Can we use gradients alone to generate a semantically meaningful input question that makes the model

Original: M generate this given answer?

Original: (without any additional assumptions on the input).

Original: More formally, if q ∈ [0 , 1] L × V is a probability distribution over all questions of length L , are gradients with respect to the question input, ∂M

Original: [Equation]
Revised: [Equation]

Original: , alone informative enough to move us towards the best human readable input that is faithful to the task being solved by M ?

Original: Our ﬁndings and the earlier argument about continuous differentiable optimization suggests this may not be feasible with current methods.

Original: To seethe correspondence to Prompt Waywardness, wecan replace D in Eq.9 with the desired outcome yand run the optimization over word distributions(cf.

Original: Footnote 10).

Original: While gradients can help guideus towards some input that makes M produce y ,such input is quite likely to not be faithful to thetask being solved by M .

Original: In the context of the aboveexample ( M being a QA system), gradients mightlead to inputs that are perhaps linguistically ﬂuent but are neither proper queries nor semanticallydescriptive of “Joe Biden” .

Original: Nevertheless, as noted earlier, gradients are stilluseful when they are applied using domain-speciﬁcconstraints.

Original: For example, one can ﬁnd local (wordlevel) perturbations that lead to a certain adversarialoutcome, if the perturbations are restricted to welldeﬁned semantic categories (e.g., “blue” can beperturbed to any other color name)

Original: (Guo et al.

Original: ,2021; Yuan et al., 2021).