Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Abstract
Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models.
It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language.
In this study, we experiment with over 30 prompt templates manually written for natural language inference (NLI).
We ﬁnd that models can learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively “good” prompts.
Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021).
That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots.
In sum, notwithstanding prompt-based models’ impressive improvement, we ﬁnd evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans’ use of task instructions.
Introduction
Suppose a human is given two sentences: “No weapons of mass destruction found in Iraq yet.”
and “Weapons of mass destruction found in Iraq.”
They are then asked to respond 0 or 1 and receive a reward if they are correct.
In this setup, they would likely need a large number of trials and errors before ﬁguring out what they are really being rewarded to do.
This setup is akin to the pretrain-andﬁne-tune setup which has dominated NLP in recent years, in which models are asked to classify a sentence representation (e.g., a CLS token) into some arbitrary dimensions of a one-hot vector.
In contrast, suppose a human is given a prompt such as:
Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “no weapons of mass destruction found in Iraq yet.”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “ ”, is it deﬁnitely correct that “weapons of mass destruction found in Iraq.”?”?”?”?”?”?”?”?”?”?”?”?”?”?”?”?”?
Then it would be no surprise that they are able to perform the task more accurately and without needing many examples to ﬁgure out what the task is.
Similarly, reformatting NLP tasks with prompts such as the underlined text above has dramatically improved zero-shot and few-shot performance over traditional ﬁne-tuned models (Schick and Schütze, 2021b; Le Scao and Rush, 2021; Sanh et al., 2021; Wei et al., 2021).
Such results naturally give rise to the hypothesis that the extra prompt text included within each input example serves as semantically meaningful task instructions which help models to learn faster, in the way task instructions help humans to learn faster.
This hypothesis is implicitly assumed by many and explicitly argued by
Mishra et al.
(2021), Schick and Schütze (2021a), and Brown et al.
While last years saw a gold rush of papers (summarized in §2) that proposed automatic methods for optimizing prompts, Logan IV et al.
(2021) compare a representative sample of these newly proposed methods and report that Schick and Schütze (2021b)’s manually written prompts still on average outperform the automatically searched prompts across a range of SuperGLUE tasks (Wang et al., 2019).
Such ﬁndings suggest that expert-crafted prompts are among the best, if not the best, which reinforces the above hypothesis that models beneﬁt from meaningful instructions.
In this paper, we test this hypothesis by evaluating various models on NLI in zero-shot and fewshot settings using more than 30 manually written templates and 13 sets of LM target words for a total of over 390 prompts.
We ﬁnd that in most cases models learn identically as fast when given irrelevant or misleading templates as they do when given instructively good templates.
Further, models ranging from 235 million to 175 billion parameters all exhibit this behavior, as do the instructiontuned models, which are trained on hundreds of manually written prompts.
While we conﬁrm Sanh et al.
(2021)’s ﬁnding that instruction tuning substantially improves the performance and robustness of prompts, we also ﬁnd that instruction-tuned models can be, in some sense, too robust and less sensitive to the semantics of the prompts, as compared to their non-instruction-tuned equivalents.
Finally, models are much more sensitive to the choice of the LM target words as opposed to the meaning of the instruction templates.
In sum, despite promptbased models’ dramatic improvement in zero-shot and few-shot learning, we ﬁnd limited evidence that models’ improvement is derived from models understanding task instructions in ways analogous to humans’ use of task instructions.
Related Work 2. Prompt-Based Models
At the time of writing, the terms “prompt tuning” and “prompting” can refer to any one or combination of three approaches described below:
Discrete Prompts reformat each example with some template text.
For example, in a sentiment analysis task, the template can be {sent} In summary, the restaurant is [prediction] , where the predicted mask word is then converted to a class prediction by a predeﬁned mapping, e.g., {“great” → positive, “terrible” → negative}.
The prompts can be manually written (Schick and Schütze, 2021a;
Bragg et al., 2021) or automatically generated (Gao et al., 2021b; Shin et al., 2020).
This approach typically tunes all parameters of the model, but its few-shot performance can exceed that of very large models (e.g., GPT-3 175B) despite using a 3 orders of magnitude smaller LM (Schick and
Schütze, 2021b;
Tam et al., 2021).
Priming (a.k.a. in-context learning) prepends k priming examples to the evaluation example, where each example is optionally wrapped in a template such as Question: {sent 1 } True or false? {label 1 } ...
Question: {sent k } True or false? {label k }
Question: {eval_sent} True or false?
[prediction] .
Notably, although models see labeled examples, their parameters do not receive gradient updates based on those examples.
Although this approach is intriguing,
Brown et al.
(2020) report that it only performs well on the largest GPT-3 model, the API of which is costly and difﬁcult to use for academic research (see Appendix B for details).
Continuous Prompts prepend examples with special tokens, optionally initialized with word embeddings; but during learning, those tokens can be updated arbitrarily such that the ﬁnal embeddings often do not correspond to any real word in the vocabulary (e.g., Lester et al., 2021; Li and Liang, 2021; Qin and Eisner, 2021).
This approach often efﬁciently tunes a much smaller set of model parameters, but these methods have not yet reported success in few-shot settings.
Moreover, foregoing prompts as expressed in natural language makes it much harder to study their semantics, and it is not clear if continuous prompts serve as task-speciﬁc instructions or simply more efﬁcient model parameters (see He et al., 2021 for a detailed analysis).
2. Analyses of Prompts
In this paper, we focus on discrete prompts because we can manually write and control their wording and semantics.
We measure the effect of prompt semantics by the model’s k -shot performance where k = { 0 , 4 , 8 , 16 , 32 , 64 , 128 , 256 } .
This setup resembles that of Le Scao and Rush (2021), but their study focuses on comparing Schick and Schütze (2021b)’s existing small set of prompts against traditional ﬁne-tuning over the training trajectories of entire training sets, whereas our study focuses on the few-shot learning trajectories among a much more diverse set of prompts designed to test speciﬁc hypotheses about the effect of prompt semantics on few-shot learning speed.
At a high-level, our ﬁndings contradict Mishra et al.
’s claim that models beneﬁt from elaborate instructions adapted from crowdsourcing annotation guides.
But note that they deﬁne “instructions” more broadly as including priming examples, and they ﬁnd that “GPT-3 beneﬁts the most from positive examples, mildly from deﬁnition, and deteriorates with negative examples.”
(p. 18).
In other words, if we ablate priming and narrow “instructions” to just the description of a task, we in fact have the same ﬁnding that instructions are only modestly beneﬁcial over no instructions (cf.
our irrelevant templates).
In a similar vein, concurrent work by Lampinen et al.
(2022) ﬁnds that other components of a prompt such as explanations of priming examples are helpful, but models are indifferent to whether the instructions in fact describe their tasks.
Finally, a growing body of concurrent work also questions the degree to which models need meaningful instructions (Khashabi et al., 2021;
Prasad et al., 2022).
One particularly noteworthy ﬁnding is that Min et al.
(2022) show that models learn just as well with incorrect labels as opposed to correct labels in priming, concluding that prompts are helping models to learn the distribution of the input text and space of possible labels (as opposed to specifying instructions of the task).
Overall Setup
We implement a manual discrete prompt modelwhich in essence is the same as that of Schick and Schütze (2021b), except their implementation includes several augmentations such as self-labeling and ensembling of multiple prompts for competitive results.
In order to focus on measuring the effect of prompts themselves, our implementation does not include those augmentations.
Following Sanh et al.
(2021) and Wei et al.
(2021), we evaluate by a rank classiﬁcation of the target words.
Baseline Model
In preliminary experiments, we ﬁne-tuned and prompt-tuned BERT, DistilBERT,
RoBERTa, ALBERT, and T5 (Devlin et al., 2019;
Sanh et al., 2019; Liu et al., 2019; Lan et al., 2020;
Raffel et al., 2020; all implemented via Wolf et al., 2020).
Conﬁrming prior work (Schick and Schütze, 2021b;
Tam et al., 2021), we ﬁnd that ALBERT consistently yields the best performance, so we use it as our baseline model.
To verify that our implementation is comparable with prior work, Figure 10 reports the RTE validation accuracy of our baseline model.
Atshots, our implementation yields a median accuracy of 70.22% (mean = 69.29%, std. dev.
= 6.3%), which is comparable to the 69.8% reported by
Schick and Schütze (2021b).
Further, Figureconﬁrms Le Scao and Rush (2021)’s ﬁnding that, while both ﬁne-tuning and prompt-tuning converge to similar results when fully trained on the entire set ( n = 2490 for RTE), prompt-tuning yields the largest improvement in the few-shot setting.
Going forward, we focus on studying the few-shot learning trajectory between 4 and 256 examples.
Instruction-Tuned Model
We additionally experiment with T0, a recently proposed instructiontuned model which is trained on over 60 datasets formatted with hundreds of manually written prompts (Sanh et al., 2021).
We experiment with both sizes of T0 (3B and 11B), as well as their noninstruction-tuned version, T5 LM-Adapted (Lester et al., 2021), as a baseline.
Very Large Model
Lastly, we experiment with the largest GPT-3 (175B) via priming (a.k.a.
incontext learning).
Although ﬁne-tuning is technically available, it is extremely limited by OpenAI’s various quotas.
See Appendix B for details on how we circumvent challenges in reproducing Brown et al.
(2020)’s results.
Data
NLI is a task where a model is asked to classify whether one piece of text (the “premise”) entails another (the “hypothesis”). We focus on NLI because all T0 variants holds out all NLI prompts and all NLI datasets in its training, which makes it a fair comparison to other models in this paper.
We use Recognizing Textual Entailment (RTE,
Dagan et al., 2006, inter alios), a series of expertannotated NLI datasets.
Speciﬁcally, we use the SuperGLUE collection of RTE (i.e., RTE1, 2, 3, and 5; all converted to binary classiﬁcation) and report their validation accuracy for comparability with prior work on prompts.
We also experiment with Adversarial NLI (ANLI, Nie et al., 2020), Heuristic Analysis for NLI Systems (HANS, McCoy et al., 2019), and
Winograd Schema Challenge (WSC, Levesque et al., 2012), reported in Appendices G.2, K, and
L, respectively. We ﬁnd no qualitative difference between their and the main RTE results except that ANLI requires much larger number of shots before obtaining any above-random accuracy, as it is designed to be a highly challenging set.
Random Seeds & Example Sampling
All experiments are run over the same set of 4 random seeds.
Within a given seed, all models see the same set of examples.
For instance, under seed 1, the 4-shot models see examples 550–553, the 8-shot models see examples 550–557, and so on.
Across different seeds, a different starting example index is drawn.
The exact training example indices are also recorded in our GitHub repository for reproducibility.
Statistical Tests
We use both ANOVA and its nonparametric equivalent, the Kruskal–Wallis test.
After ﬁnding a signiﬁcant difference among multiple categories of templates, we report pairwise signiﬁcance with the independent two-sample t -test and the Wilcoxon rank-sum test.
We set α = 0 .and
apply the Bonferroni correction to account for multiple comparisons.
For all results reported in this paper, both t -test and Wilcoxon agree.  Effect of Templates
Our research question is whether models understand prompts as meaningful task instructions analogous to how humans would.
For intuition, suppose an experimenter provides a human annotator with an informative instruction of a reasonably easy task.
If the annotator understands the instruction, we expect them to perform better than when the experimenter provides intentionally misleading instructions, makes irrelevant chitchat, or says nothing at all. Accordingly, we write various prompt templates that correspond to these different scenarios and evaluate models’ performance with these templates in zero-shot and few-shot settings. 4. Method We write 5 categories of templates (Table 1), with at least 5 templates for each category (10 for instructive):
• Instructive: how we would describe the NLI task to a human who has never seen this task before.
• Misleading-Moderate: instruct the models to perform a task related or tangential to NLI such that, if the model were to perform the task as explicitly instructed, it would perform poorly on NLI in general.• Misleading-Extreme: instruct the models to perform a task unrelated to NLI. • Irrelevant: concatenate the premise, a sentence unrelated to any NLP task, and the hypothesis. • Null: concatenate the premise and the hypothesis without any additional text. [Table]
See Table 1 for examples and Appendix F for the full list. We use “prompt” to mean a unique combination of a template and a predeﬁned LM target word for each class label. For example, {“yes” → entailment, “no” → non-entailment} are the default targets for the template {premise} Should we assume that {hypothesis}? [prediction] .
In this section, to control for the effect of target words, a template’s performance is always reported with “yes”/“no” as its target words, which consistently perform best.
In Section 5, we control for the templates and study the effect of different target words. We further control for punctuation, declarative vs. interrogative templates, and the order of concatenation (always {premise} some template text {hypothesis}[prediction] ). After preliminary experiments, to avoid cherry picking, all prompts reported in this paper were written prior to evaluation, i.e., we do not allow retroactively editing prompts for performance manipulations, except for an ablation study that explicitly studies the effect of punctuation (Appendix A). 4. Result Irrelevant Templates
We ﬁnd that models trained with irrelevant templates learn just as fast as those trained with instructive templates, with no practical difference 3 at any number of shots (Figure 1). This is true for all models and all datasets in our experiments, including the largest GPT- (Figure 2).
[Figure]
irrelevant
[Figure]
R TE V a li d a ti on A cc u r ac y
[Figure]
Misleading
Templates
There is no consistent relation between the performance of models trained with templates that are moderately misleading (e.g. {premise} Can that be paraphrased as "{hypothesis}"? ) vs. templates that are extremely misleading (e.g., {premise} Is this a sports news? {hypothesis} ).
T0 (both 3B and 11B) perform better given misleading-moderate (Figure 3), ALBERT and T5 3B perform better given misleading-extreme (Appendices E and G.4), whereas T5 11B and GPT-3 perform comparably on both sets (Figure 2; also see Table 2 for a summary of statistical signiﬁcances.)
Despite a lack of pattern between the two misleading categories, however, it is       0.
instructive misleading-moderate misleading-extreme
Number of Shots
R TE V a li d a ti on A cc u r ac y consistent
that each model exhibits signiﬁcantly better performance on instructive templates compared to at least one category of misleading templates.
Null Templates
Models trained with null templates perform far worse than all other categories of templates (see Appendix G for all null results).
Here, we focus on ALBERT (an encoderonly masked language model), which allows more permutation of concatenation orders by placing mask in the middle of sentences.
We see that, although null templates are much worse in aggregate, some subset of them (e.g., {premise} [mask] {hypothesis} ) are still able to learn nearly as fast as the average instructive template aftershots (Figure 13).
Zero-Shot
So far, we have focused on few-shot results. At zero shots, all models (including GPT175B) perform only marginally above random, except the instruction-tuned T0.
Thus, for our analysis of zero shot performance, we focus on T0.
Figureshows that there is no practical difference between the performance of T0 3B given instructive templates and either category of misleading templates.
T0 11B performs better, although it also shows no practical difference between misleading-moderate and instructive templates.
Lastly, T0++ (trained on more datasets than other T0 variants), is the only model in this paper that shows statistically signiﬁcantly different performance across all categories of prompts.
However, there remains the caveat that it still performs arguably too well in absolute terms with pathological prompts, which we discuss in the next section.
[Table]
(cid:81)(cid:88)(cid:79)(cid:79)
given… is it guaranteed true that does the paragraph start with “the” is this grammatically correct are there lots of similar words is the sentiment positive inflections are annoying are we justified in saying that size #shots inst.
> mis-moderate inst.
> mis-extreme inst.
>
irrelevant inst. >
null
T 3B (cid:51)
T 11B (cid:51) (cid:51) (cid:51)
T0++ 11B (cid:51) (cid:51) (cid:51) (cid:51)
ALBERT 235M 4 -(cid:51) (cid:51)
T5 LMA 770M 4T5 LMA 3B 4 -(cid:51) (cid:51)
T 3B 4 -(cid:51)
(cid:51)
T5 LMA 11B (cid:51) (cid:51) (cid:51)
T 11B (cid:51) (cid:51)
T0++ 11B (cid:51) (cid:51)
GPT175B (cid:51)
4. Discussion
Recall that a common assumption in the literature is that prompts require experts to clearly and correctly describe the task at hand (§1).
In contrast,
Table 2 summarizes that, with the exception of
T0++ at zero shots, all models perform essentially as well with some pathological prompts as they do with proper prompts. Notably, despite being much larger than its competitors, GPT-3 shows the same patterns of behaviors, suggesting that mere scaling does not address this issue. Meanwhile, the evidence from instruction tuning is mixed. Although
Sanh et al.
(2021) are right that instruction tuning yields substantial improvement in performance as well as robustness as measured by variance, T0 is somewhat too robust and less sensitive to the semantics of the prompts in terms of distinguishing proper instructions from pathological ones, compared to T5 of the same size in the few-shot setting (Figure 2).
In the zero-shot setting, we do see that that the largest model instruction-tuned with the most datasets (T0++) improves a model’s sensitivity to prompt semantics.
This is a positive result, but it comes with the caveat that there still exist numerous examples of pathological prompts that perform just as well as the proper ones do.
To be charitable to randomness in neural models, we hold this study to a higher standard by comparing means and medians among categories with statistical tests.
yes;no agree;disagree
no;yes cat;dog
Number of Shots
R TE V a li d a ti on A cc u r ac y
Nevertheless, for our research question, existence proofs alone are still alarming.
For example, without any gradient update nor priming, it is striking that out-of-the-box T0++ scores a high accuracy of 78% with the extremely misleading {premise} Is that grammatically correct?
{hypothesis} , the same accuracy as it achieves with a proper instruction {premise} Are we justified in saying "{hypothesis}"?
If models were truly classifying whether the text is grammatical, it would have only scored 52.7% because RTE is written by experts and all examples are grammatical.
Even templates that underperform the instructive ones seem to be too good.
For example, it is difﬁcult to imagine a human scoring 72% zero-shot with the prompt {premise} Inflections are annoying and thank god that
Middle English got rid of most of them.
{hypothesis} for a nuanced task like
NLI.  Effect of Target Words 5. Method In this experiment, we study the effect of different LM target words given a ﬁxed template.
We write 4 categories of targets, with at least 3 pairs of target words for each category (except the singleton yesno category):
Yes-no: Model is expected to predict the word “yes” for entailment and “no” for nonentailment.
Yes-no-like: Semantically equivalent to yesno but using superﬁcially different words, e.g., “true”/“false”, “positive”/“negative”.
{prem} Based on the previous passage, is it true that "{hypo}"?
{“cat” → entailment, “dog” → non-entailment} {prem} Does the paragraph start with "the"? {hypo} {“yes” → entailment, “no” → non-entailment}
Number of Shots
R TE V a li d a ti on A cc u r ac y
Arbitrary: Model is expected to predict arbitrary words that have no semantic relation to the entailment task, e
g., “cat” for entailment, “dog” for non-entailment.
Reversed:
Model is expected to predict the opposite of the (intuitive) yes-no and yes-nolike labels, e g., “no” for entailment, “yes” for non-entailment. See Appendix F.3 for the full list. Within the arbitrary category, in addition to the common anglophone ﬁrst names as Le Scao and Rush (2021) use, we also include word pairs with high semantic similarity, low similarity, and pairs which are highly frequent in the English language, but we ﬁnd no consistent difference among these various subcategories of the arbitrary category.
5. Result
For both ALBERT and T0, we ﬁnd that models trained with yes-no targets learn a good deal faster than those trained with yes-no-like targets and dramatically faster than those with arbitrary and reversed targets.
For example, Figure 5 shows the top-performing instructive template trained with different target words.
At 32 shots, the difference between the median accuracies of “yes”/“no” vs.
“no”/“yes” is 22 .
2% , far larger than the effect size of varying categories of templates in Section 4.
Aggregating over all combination of templates and targets, Figure 16 conﬁrms that the choice of target words matter much more than the meaning of the templates.
5. Discussion
The fact that models consistently learn slower with arbitrary and reversed target words is a positive result: this type of performance differential is consistent with what we expect for models that are correctly sensitive to the semantics of the words.
However, there are several important negative results in these experiments as well.
First, the effect of the target words overrides the semantics of the overall prompt.
Consider two kinds of templatetarget combinations:
An irrelevant or misleading template + yes-no targets, e.g., {
Does the paragraph start with "the"?
[yes/no] {hypothesis}
An instructive template + arbitrary targets, e.g., {premise} Based
on the previous passage, is it true that "{hypothesis}"? [cat/dog]
Figure 6 shows that combinations such as (1) often dramatically outperform (2).
However, (2) simply requires ﬁguring out a mapping: “Reply ‘cat’ if entailed and reply ‘dog’ if not entailed”.
For humans, this can be learned in a few shots, e.g., Ferrigno et al.
(2017) showed that adults can reach 60% accuracy in 18 trials 4 for an arbitrary map of {more numerous → star shape, less numerous → diamond shape} without receiving any language instructions.
In contrast, models under many arbitrary LM targets struggle to reach 60% median accuracy even by 64 shots with instructive templates (Figuregreen; Figure 5 red, purple).
Further, even given intuitive yes-no-like targets such as “agree”/“disagree” and “good”/“bad”, models learn much slower compared to when given “yes”/“no”.
As Figure 5 (green vs. dark green) and
Figure 16 (ﬁrst vs. second x-axis group) show, there exists a large performance gap between yes-no and yes-no-like targets which is not closed untilshots.
Moreover, when we try to help the models by appending target hints such as “True or false?”
to the templates, performance often drops instead, echoing Sanh et al.
(2021) and Wei et al.
(2021)’s ﬁndings that including answer choices in input sequence make models perform worse for certain tasks.  General Discussion 6. Summary and Interpretation Our main research question is whether models understand prompts as meaningful task instructions analogous to how humans would. Again, suppose an experimenter provides a human annotator with an informative instruction of a reasonably easy task.
If the annotator understands the instruction, we expect them to perform better than when the experimenter provides misleading instructions, irrelevant instructions, or no instructions at all.
Sectionshows that the performance of most models is insensitive to the difference between instructive and irrelevant templates, moderately sensitive between instructive and misleading templates, and highly sensitive between instructive and null templates.
Comparing to the effect of the templates, however, Section 5 shows that models are much more sensitive to the semantics of the target words: they learn far slower with arbitrary or reversed target words as desired.
However, they are overly sensitive to semantically equivalent yes-no-like words (i.e., performing much worse with “agree”/“disagree” than with “yes”/“no”), and the choice of target words override the semantics of the templates (e.g., performing much better given a irrelevant template with “yes”/“no” targets than with an instructive template with arbitrary targets such as “cat”/“dog”).
Our main argument throughout the paper shares the same logic as a recent line of studies (Sinha et al., 2021; O’Connor and Andreas, 2021; Pham et al., 2021; Gupta et al., 2021) which argue that the fact that LMs achieve good performance under ideal conditions is insufﬁcient to establish language understanding because they also succeed under pathological conditions (e.g., sentences with shufﬂed word order) where humans fail catastrophically.
In other words, the fact that models are so good at inferring the gold labels from pathological inputs casts major doubts on whether models make inferences in any way that resembles how humans make inferences.
For our results, the fact that models are so good at learning from patho- logical instructions likewise casts major doubts on whether models understand prompts as instructions in any way that resembles how humans understand instructions.
6. Alternative Interpretations and Future Directions
As with any extrinsic evaluation, accuracy cannot directly measure understanding.
For example, a human could perfectly understand an instruction but still, e.g., have the same accuracy with instructive vs. irrelevant templates because the task itself is too hard (a lack of competence) or because they for some reason ignore the instructions (a lack of compliance).
We discuss these two possibilities below.
Lack of Competence
This is primarily a concern for non-instruction-tuned models at zero shots, where all models perform only slightly above random, and thus a lack of statistical signiﬁcance among template categories is ambiguous as to whether models lack understanding of NLI instructions vs. if models lack the competence in NLI per se.
This is why our study largely focuses on the fewshot setting, where a lack of competence is less of a concern, as models do competently achieve good accuracies that are only moderately below the stateof-the-art non-few-shot models.
Another counterargument is that maybe no models ever actually reason about if a premise entails a hypothesis.
Maybe they just always exploit spurious or heuristic features and, if only they were competent in properly reasoning about entailment relations, then the meaning of NLI instructions would matter.
This argument is possible, although, ﬁrst, it hinges on to what extent NLI (or any other behavioral evaluation) can measure language understanding, which is a complex debate beyond the scope of this paper.
Second, in preliminary experiments (Appendix K), our models actually zero-shot transfer reasonably well to HANS (McCoy et al., 2019), a dataset designed to diagnoses models use of NLI heuristics.
Thus, it is unlikely that models are entirely incompetent in reasoning about entailment relations and solely rely on heuristics.
Regardless, further differentiating competence in understanding task instructions vs. competence in tasks per se is an important direction for future work.
Lack of Compliance
Another interpretation is that irrelevant prompts perform the same as the instructive ones because models simply ignore the prompts altogether.
However, a lack of compliance alone cannot explain our results.
If models truly ignore the prompts, we should not see any systematic differences between any categories of prompts.
Instead, we do see consistent patterns that instructive and irrelevant templates make models learn signiﬁcantly faster than misleading and null templates do (Table 2).
A more nuanced counterargument is that although models do not ignore their prompts entirely, perhaps it “takes less effort” for models to use the spurious or heuristic features for predictions as opposed to the more complex syntactic or semantic features (Lovering et al., 2021; Warstadt et al., 2020) required to properly comply with the instructions.
However, spurious features alone likewise cannot explain the observed performance gaps.
Recall that, within each random seed, all models see exactly the same training examples (with the same spurious features).
Thus, to the extent that models perform differently with some prompts compared to others, it may be due to some complex interactions between the (spurious or semantic) features in prompts and the spurious features in data examples.
One possible example of this interaction is that punctuation has a large effect for irrelevant templates, but instructive templates seem to be able to suppress such effect (Appendix A).
Investigating the nature of this interaction is a promising direction for future work, and it suggests a way in which the semantics of the prompt might matter, e.g., by affecting the models’ inductive biases, even if models do not interpret or use the instructions in the same way as humans would.
Conclusion
In this study, we train several models with over 30 manually written templates and 13 sets of LM targets for NLI.
We ﬁnd that models often learn equally fast with misleading and irrelevant templates as they do with instructive ones, and that the choice of the target words overrides the meaning of the overall prompts.
Although models do not entirely ignore the meaning of the prompts, our results contradict a hypothesis commonly assumed in the literature that models use prompts as semantically meaningful task instructions in ways analogous to humans’ use of instructions.
[-Conclusion 
-]
[-A Effect of Target Words 
-]
[-B Effect of Punctuation 
-]
[-C Details and Lessons from Experimenting with GPT-3’s API 
-]
[-C.3 Other Tips for Working with GPT-3 
-]
Ethical Considerations
The fact that even the largest LMs appear to follow yet do not actually follow users’ instructions has important implications, especially considering the increasing commercial use of LMs.
While traditional ﬁne-tuned models also pose challenges in interpretability, with prompt-based models, an illusion of instruction following can be more pernicious than having no instructions at all.
The intuitive interface that prompts provide might make them more accessible to lay users, and can mislead users to think that their instructions are being understood and followed.
Our results suggest that cautions are needed even more than they were with traditional ﬁne-tuned models.
Acknowledgments
We are grateful to Colin Raffel, Victor Sanh, Sasha
Rush, Stephen Bach, Roman Feiman, Teven Le
Scao, Ian Tenney, Dan Garrette, Jason Wei, Satoshi
Sekine, Mike Tien-Chien Chiang, Xavier Fontaine,
Pierre Colombo, Ryan Teehan, Debajyoti Datta,
William Rudman, Ruochen Zhang, Daniel Cohen,
George Zerveas, Eric Rosen, Kaiyu Zheng, Nihal
Nayak, Roma Patel, Charles Lovering, Tian Yun,
Jack Merullo, and Aaron Traylor for comments and discussions on early drafts of this paper.
Special thanks to Victor, Colin, and Teven for technical clariﬁcations and code review.
Furthermore, Albert is indebted to Colin and
Sasha for their patience on the many iterations of the zero-shot Figure 4 as well as invaluable mentorship throughout the T0 project.
A Effect of Punctuation
For irrelevant templates, we ﬁnd a large effect from the use of quotation and question marks in templates.
It is natural to write such punctuation in instructive templates as they help humans to parse an NLI hypothesis as an embedded clause within an instruction sentence (e.g.,
Given {premise} Should we assume that "{hypothesis}" is true? ).
For control, we also use quotation and question marks (“qmarks” hereafter) in irrelevant templates where they would not have made sense naturally, e.g., {premise} Single-family zoning is bad for American cities. [Figure]
R TE V a li d a ti on A cc u r ac y
"{hypothesis}"?
As an ablation, when we remove these qmarks from irrelevant templates, the performance of ALBERT and T0 drops substantially (Figures 7 and 8).
In contrast, for T5, qmarks make no difference for irrelevant templates; yet, removing qmarks from instructive templates—where qmarks are natural—boosted performance instead for T5 (Figure 9), but not for
T0 nor ALBERT.
Additionally, as a coincidence, most misleading templates contain both quotation and question marks, while most misleading-far templates contain only question marks (Appendix F).
But as noted in Section 4.2, there is no consistent pattern between those two misleading categories.
In other words, punctuations alone cannot explain everything.
As discussed in Section 6.2, the full explanation is likely a combined interactions between the spurious features and the semantics of the templates.
Lastly, note that Schick and Schütze (2021b) and many subsequent papers’ prompts for
NLI (e.g., "{hypothesis}" ?
| [mask]. "{premise}" ) are basically null templates with some variation in punctuation between the hypothesis and the premise.
We ﬁnd that models learn poorly with the vanilla {hypothesis} [mask] {premise} , but they learn as fast as the instructive templates with Schick & Schütze’s punctuated version.
That being said, note again that punctuation alone cannot explain the performance gap, since models trained with [mask] {hypothesis} {premise} (Fig- ure 13, pink) perform second to best, yet swapping their premises and hypotheses (Figure 13, purple) makes it the worst performing among all null templates.
B Details and Lessons from Experimenting with GPT-3’s API B. Choice of Model
We use the davinci model provided by OpenAI
LP’s API, which corresponds to 6 the 175 billion parameter model reported in Brown et al.
Concurrent to our work, OpenAI released a new product called the “Instruct Series”, but we decided to not experiment with the Instruct Series because no academic paper or technical documentation of any kind is available with the Instruct Series at the time of writing aside from the following claim on their website:
The Instruct models share our base
GPT-3 models’ ability to understand and generate natural language, but they’re better at understanding and following your instructions.
You simply tell the model what you want it to do, and it will do its best to fulﬁll your instructions.
This is an important step forward in our goal of building safe models that are aligned with human interests.
Crucially, the Instruct Series is inappropriate for reproducible research because it is unknown what datasets and prompts these models are trained on, and whether any task categories are systematically held out as done by Sanh et al.
(2021) and Wei et al.
If it is trained on any prompt or dataset of
NLI, it would not be zero-shot, making it an unfair comparison to other models in our experiments.
Second, it is still in beta and its training, held-out, and prompt mixtures could change.
At least two
Instruct Series models were made available in sequence during our writing, and it is not clear if we experiment on an older version, whether it will still be available and reproducible in the future.
B. Priming vs. Fine-Tuning
As mentioned in Section 3, we use priming (a.k.a. in-context learning) in lieu of ﬁne-tuning because, at the time of writing, OpenAI’s ﬁne-tuning API is limited to 10 runs per month.
To train 30 prompts at only two number of shots would take 6 months, assuming we get hyperparameters right at ﬁrst try.
Further, each training run is limited to a maximum of 5 epochs, which often entails an insufﬁcient number steps for few-shot training.
We were unable to ﬁne-tune GPT to any reasonable accuracy with our allowed 10 tries in the ﬁrst month.
Finally, the ﬁne-tuning API is limited to GPT variants up to 6.7B, not the 175B model we plan to experiment with.
With priming, we are able to reproduce Brown et al.
(2020)’s
zero-shot performance on RTE but only with their exact prompt reported in their Figure G.31, all other (even instructive) prompts perform at random at zero shots, suggesting that their reported prompt is highly cherry-picked.
We are unable to reproduce their reported few-shot result because they report it at 32 shots, but their API only permits a context length up to 2049 tokens, which is insufﬁcient for RTE.
We ﬁnd that 16 shots are the highest one can reach within the token limit.
Like the gradient updated models, we document the exact examples we use for few-shot priming in our GitHub repository.
Unlike the gradient updated models, which are trained on the same k examples, priming models use different sets of k priming examples for each inference example (Brown et al., 2020, p. 20).
This means that GPT’s performance reﬂects the fact that, overall, it has seen far more than k examples, making it not directly comparable to the few shots of the gradient updated models.
This is not ideal, but our GPT few-shot performance already underperforms what Brown et al.
(2020) report, so we choose to not further restrict it to have the same ﬁxed priming examples for all inference examples, which could run into a lack of competence issue (§6.2) that make its results unusable for our research question.
Lastly, unlike the gradient updated models, we do not run multiple seeds with our GPT experiments because, ﬁrst, they are expensive.
As the
API bills by token, using k shots of priming example effectively multiplies the total cost by k .
Sec- ond, OpenAI imposes a monthly quota for each lab, so running multiple seeds will take several more months to complete.
B. Other Tips for Working with GPT-
Using the logprobs argument in their API, we obtain the top 99 predicted target word and their log probabilities.
Following Sanh et al.
(2021) and Wei et al.
(2021), we evaluate by a rank classiﬁcation of the target words, i.e., if the gold target word is “yes”, we consider it as correct as long as the probability of “yes” is higher than that of “no”, regardless of whether “yes” is the top-1 prediction generated by the model.
Alarmingly, we ﬁnd that these top-99 predictions are semantically inconsistent ranked, e.g., for one data example and its top-99 word predictions, it is often the case that, e.g., P(yes) > P(no) but P(Yes) < P(No).
Thus, the choice of the target words’ surface form makes a substantial difference in the overall performance.
(Not to mention the problem of choosing between yes/no, true/false, correct/incorrect, etc.
as studied in Section 5.)
OpenAI recommends having no trailing space in the input and let the model predict the ﬁrst token with a leading space as in “ Yes”.
We ﬁnd that although stripping the leading space sometimes leads to higher performance for some prompts, overall not applying stripping or other token normalization performs the best.
Another point researchers should pay attention to is the use of what OpenAI calls a “separator” inserted between priming examples.
In preliminary experiments, we initially use newline characters as appeared in Brown et al.
(2020)’s
Appendix G. We later discover that OpenAI recommends using ### or \n###\n as separators.
We use the latter and ﬁnd consistent performance improvement over just using newline characters, and we use it throughout in our main experiments.
C Hyperparameters
For encoder-only models, we follow Schick and
Schütze (2021b) and Le Scao and Rush (2021)’s recommendations and use a learning rate of 1 e − 5 .
For T5 and T0 models, we follow Raffel et al. (2020) and Sanh et al.
(2021)’s recommendations and use a learning rate of 1 e − 4 .
We run several preliminary experiments with learning rates ( 3 e − 4 , 1 e − 4 , 5 e − 5 , 1 e − 5 ) deviating from their recommendations and they perform worse, although our search is not exhaustive due to the high cost of running multiple prompts with multiple random seeds.
Note that T5 and T0 are trained with the Adafactor optimizer (Shazeer and Stern, 2018) in Mesh
TensorFlow.
Our implementation is in PyTorch, and we ﬁnd that ﬁne-tuning T5 with PyTorch’s implementation of Adafactor yields substantially worse results than the usual choice of the AdamW optimizer.
We corresponded with Raffel et al.
(2020), who advised us that it might be due to the fact that
PyTorch does not have the same learning rate scheduler implementation as TensorFlow’s Adafactor does.
They recommended us to simply use AdamW, which is what we did.
This is somewhat unfortunate because Adafactor is much more memory efﬁcient, which would have drastically reduced the compute resources required and thus enable more comprehensive experiments of the 11B models, which are currently limited to 0 shots and 16 shots only.
Although most models seem to obtain the highest validation accuracy at very early epochs, we train all models to 30 epochs (20 epochs for 11B models) to be safe and select the checkpoint with the highest validation accuracy.
All models use a batch size of 4 with 4 gradient accumulation steps for an effective batch size of 16.
Note that because we use a rank classiﬁcation of single-token target words, decoding sampling methods (e.g., beam search, top- k , top- p ) are unnecessary.
We follow Raffel et al.
and add EOS tokens for input sequences, which yields higher fewshot performance compared to not adding EOS as done by Sanh et al.
However, we omit EOS in the zero-shot setting, which exactly reproduces the results reported by Sanh et al.
See T0’s
GitHub repository readme 10 for more information.
D Compute Used
Each ALBERT 235M model is trained on a single
Nvidia RTX3090.
Their main experiments took approximately 192 GPU hours.
Each T5 LMA 770M model is trained on a single A6000.
Their main experiments took approximately 48 GPU hours.
The 3B models are each trained by partitioning their layers over four RTX3090s.
T5 and T0’s main experiments took approximately 2,304 GPU hours in total.
The 11B models are each trained on eight V100s (each with 32GB of memory).
T5, T0, and T0++’s main experiments took approximately 1,728 GPU hours in total.
(Due to their large GPU memory requirement, we were only able to complete one number of shots.)
E Additional Figures Discussed in the Main Text
[Figure] R TE V a li d a ti on A cc u r ac y [Figure]
[Figure]
R TE V a li d a ti on A cc u r ac y
[Figure]
R TE V a li d a ti on A cc u r ac y
[Figure]
[Figure]
[Figure]
yes;no good;bad
no;yes cat;dog
Number of Shots
R TE V a li d a ti on A cc u r ac y
[Figure]
R TE V a li d a ti on A cc u r ac y
[Figure]
R TE V a li d a ti on A cc u r ac y
F All Prompts F. Main Experiment Templates
[Table]
Williams et al.
(2018, p. 3) [Table] Brown et al. (2020, p. 59)
[Table]
justiﬁed in
[Table]
[Table] [Table] [Table] } Should we assume that "{hypothesis}" is true?
[Table]
{premise} Based on the previous passage, is it true that "{hypothesis}"?
[Table]
{premise} Is it guaranteed true that "{hypothesis}"?
[Table]
Suppose {premise} Can we infer that "{hypothesis}"?
[Table]
Given that {premise} Does it follow that "{hypothesis}"?
[Table]
premise} Question: Does this imply that "{hypothesis}"?
[Table]
Given that {premise} Therefore, it must be true that "{hypothesis}"?
misleading-moderate {premise} Do most of the above words appear in the following passage?
{hypothesis} misleading-moderate {premise} Are there lots of similar words in "{hypothesis}"?
misleading-moderate {premise} Does that have the same meaning as "{hypothesis}"?
misleading-moderate {premise} Can that be paraphrased as: "{hypothesis}"?
misleading-moderate {premise} Can that be summarized as "{hypothesis}"?
misleading-extreme {premise} Does the paragraph start with "the"? {hypothesis} misleading-extreme {premise} Is this grammatically correct?
{hypothesis} misleading-extreme {premise} Is the sentiment positive?
{hypothesis} misleading-extreme {premise} Is this a sports news?
{hypothesis} misleading-extreme {premise} Is this French?
{hypothesis} irrelevant {premise} Single-family zoning is bad for American cities. "
{hypothesis}"?
irrelevant {premise} Inﬂections are annoying and thank god that Middle English got rid of most
[Table]
[Table]
Greenberg (1974, p. 141)
[Table]
Tsuji and Sutherland (1980, p. 148)
[Table]
Plato (c. 399 BC, 10a)
[Table]
Of course this research question became moot as we found that cannot even pathological
F
[Table]
Ablation Experiment Templates
[Table]
instructive
[Table]
qmarks {premise} Using only the above description and what you know about the world, {hypothesis}is deﬁnitely correct.
Yes or no instructive sans qmarks {premise} \ nquestion: {hypothesis}Yes or no \ nanswer: instructive sans qmarks {premise} Are we justiﬁed in saying that {hypothesis} instructive sans qmarks Given {premise} Should we assume that {hypothesis}is true instructive sans qmarks {premise} Based on the previous passage, is it true that {hypothesis} instructive sans qmarks Given {premise} Is it guaranteed true that {hypothesis} instructive sans qmarks Suppose {premise} Can we infer that {hypothesis} instructive sans qmarks Given that {premise} Does it follow that {hypothesis} instructive sans qmarks {premise} Question: Does this imply that {hypothesis} instructive sans qmarks Given that {premise} Therefore, it must be true that {hypothesis} irrelevant sans qmarks {premise} Single-family zoning is bad for American cities.
{hypothesis} irrelevant sans qmarks {premise} Inﬂections are annoying and thank god that Middle English got rid of most of them.
{hypothesis} irrelevant sans qmarks {premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he arrived at the same results 30 years ago.
{hypothesis} irrelevant sans qmarks {premise} If bonito ﬂakes boil more than a few seconds, the stock becomes too strong.
{hypothesis} irrelevant sans qmarks {premise} Is the pious loved by the gods because it is pious.
Or is it pious because it is loved by the gods.
{hypothesis}
F. All Target Words
[Table]
G
Aggregated Results
G
ALBERT on RTE
[Table]
Number of Shots
R TE V a li d a ti on A cc u r ac y
[Table] [Table]
Number of Shots
R TE V a li d a ti on A cc u r ac
[Table]
on
[Table]
[Table] [Table]
[Table]
Number of Shots
R TE V a li d a ti on A cc u r ac y
[Table]
G.
T5 3B on RTE
[Table]
Number of Shots
R TE V a li d a ti on A
[Table]
u
[Table]
G.
T0 3B on RTE
[Table]
Number of Shots
R TE V a li
[Table]
a ti on A
[Table]
G.
T0 3B on ANLI R
[Table]
Number of Shots
R TE V a li d a ti on A cc u
[Table]
ac [Table] G.  T5 11B, T0 11B, and GPT-3 175B (Figure 2)
[Table]
H Results of Individual Templates
H.
ALBERT       0.
aggregated instructive templates
{premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? "
{hypothesi {premise} Inflections are annoying and thank god that Middle English got rid of most of them. "
{hy {premise} Is the pious loved by the gods because it is pious?
Or is it pious because it is loved by the {premise} Single-family zoning is bad for American cities. "
{hypothesis}"? {mask} {premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
[Figure]
R TE V a li d a ti on A cc u r ac y
aggregated instructive templates
{premise} Are there lots of similar words in "{hypothesis}"?
{mask} {premise} Can that be paraphrased as: "{hypothesis}"?
{mask} {premise} Can that be summarized as "{hypothesis}"?
{mask} {premise} Do most of the above words appear in the following passage?
{hypothesis} {mask} {premise} Does that have the same meaning as "{hypothesis}"?
{mask}
[Figure]
R TE V a li d a ti on A cc u r ac y
aggregated instructive templates
{premise} Does the paragraph start with "the"? {hypothesis} {mask} {premise} Is the sentiment positive?
{hypothesis} {mask} {premise} Is this French?
{hypothesis} {mask} {premise} Is this a sports news?
{hypothesis} {mask} {premise} Is this grammatically correct?
{hypothesis} {mask}
Number of Shots
R TE V a li d a ti on A cc u r ac y
Given that {premise} Does it follow that "{hypothesis}"?
{mask}
Given that {premise} Therefore, it must be true that "{hypothesis}"?
{mask}
Given {premise} Is it guaranteed true that "{hypothesis}"? {mask}
Given {premise} Should we assume that "{hypothesis}" is true?
{mask}
Suppose {premise} Can we infer that "{hypothesis}"?
{mask} {premise} question: {hypothesis}
Yes or no?
answer: {mask} {premise} Are we justified in saying that "{hypothesis}"?
{mask} {premise} Based on the previous passage, is it true that "{hypothesis}"?
{mask} {premise} Question: Does this imply that "{hypothesis}"? {mask} {premise} Using only the above description and what you know about the world, "{hypothesis}" is d
Number of Shots
R TE V a li d a ti on A cc u r ac y
H.
T0 (3B)       0.
aggregated instructive templates
{premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? "
{hypothesi {premise} Inflections are annoying and thank god that Middle English got rid of most of them. "
{hy {premise} Is the pious loved by the gods because it is pious?
Or is it pious because it is loved by the {premise} Single-family zoning is bad for American cities. "
{hypothesis}"? {premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
[Figure]
R TE V a li d a ti on A cc u r ac y
aggregated instructive templates
{premise} Are there lots of similar words in "{hypothesis}"?
{premise} Can that be paraphrased as: "{hypothesis}"? {premise} Can that be summarized as "{hypothesis}"? {premise} Do most of the above words appear in the following passage?
{hypothesis} {premise} Does that have the same meaning as "{hypothesis}"?
Number of Shots
R TE V a li d a ti on A cc u r ac y
aggregated instructive templates
{premise} Does the paragraph start with "the"? {hypothesis} {premise} Is the sentiment positive?
{hypothesis} {premise} Is this French?
{hypothesis} {premise} Is this a sports news?
{hypothesis} {premise} Is this grammatically correct?
{hypothesis}
Number of Shots
R TE V a li d a ti on A cc u r ac y
Given that {premise} Does it follow that "{hypothesis}"?
Given that {premise} Therefore, it must be true that "{hypothesis}"?
Given {premise} Is it guaranteed true that "{hypothesis}"?
Given {premise} Should we assume that "{hypothesis}" is true?
Suppose {premise} Can we infer that "{hypothesis}"? {premise} question: {hypothesis}
Yes or no?
answer: {premise} Are we justified in saying that "{hypothesis}"?
{premise} Based on the previous passage, is it true that "{hypothesis}"?
{premise} Question: Does this imply that "{hypothesis}"? {premise} Using only the above description and what you know about the world, "{hypothesis}" is d
Number of Shots
R TE V a li d a ti on A cc u r ac y
H.
T5 LM-Adapted (3B)       0.
aggregated instructive templates
{premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? "
{hypothesi {premise} Inflections are annoying and thank god that Middle English got rid of most of them. "
{hy {premise} Is the pious loved by the gods because it is pious?
Or is it pious because it is loved by the {premise} Single-family zoning is bad for American cities. "
{hypothesis}"? {premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
[Figure] of Shots
R TE V a li d a ti on A cc u r ac y
aggregated instructive templates
{premise} Are there lots of similar words in "{hypothesis}"?
{premise} Can that be paraphrased as: "{hypothesis}"? {premise} Can that be summarized as "{hypothesis}"? {premise} Do most of the above words appear in the following passage?
{hypothesis} {premise} Does that have the same meaning as "{hypothesis}"?
Number of Shots
R TE V a li d a ti on A cc u r a c y
aggregated instructive templates
{premise} Does the paragraph start with "the"? {hypothesis} {premise} Is the sentiment positive?
{hypothesis} {premise} Is this French?
{hypothesis} {premise} Is this a sports news?
{hypothesis} {premise} Is this grammatically correct?
{hypothesis}
Number of Shots
R TE V a li d a ti on A cc u r a c y
Given that {premise} Does it follow that "{hypothesis}"?
Given that {premise} Therefore, it must be true that "{hypothesis}"?
Given {premise} Is it guaranteed true that "{hypothesis}"?
Given {premise} Should we assume that "{hypothesis}" is true?
Suppose {premise} Can we infer that "{hypothesis}"? {premise} question: {hypothesis}
Yes or no?
answer: {premise} Are we justified in saying that "{hypothesis}"?
{premise} Based on the previous passage, is it true that "{hypothesis}"?
{premise} Question: Does this imply that "{hypothesis}"? {premise} Using only the above description and what you know about the world, "{hypothesis}" is d
Number of Shots
R TE V a li d a ti on A cc u r ac y
[Table]
PHM 0.
[Table]
null
[Table]
_HPM 0.
[Table]
MNLI_YN 0.
[Table]
GPT_YN 0.
[Table]
justiﬁed_in_saying 0.
[Table]
should_assume 0.
[Table]
is_it_true 0.
[Table]
guaranteed_true 0.
[Table]
can_we_infer 0.
[Table]
does_it_follow 0.
[Table]
instructive does_this_imply 0.
[Table]
modal_be_true 0.8087 T
[Table]
-moderate
words_appear 0.
[Table] [Table]
-moderate similar_words 0.7329 T
[Table] [Table]
-moderate same_meaning 0.7545 T
[Table]
11B)
[Table]
-moderate paraphrase 0.7617 T
[Table]
) misleading-moderate summarize 0.
[Table] [Table]
-extreme start_with_the 0.
[Table]
-extreme grammatical 0.
[Table] [Table]
sentiment 0.
[Table]
-extreme sportsball 0.
[Table]
-extreme french 0.
[Table]
irrelevant zoning 0.
[Table]
irrelevant gauss 0.
[Table]
irrelevant katsuobushi 0.
[Table]
inﬂection
[Table]
J
Comparison of LM targets, Controlling for the Template
[Figure]
R TE
V a li d a ti on
A cc u r ac y
[Figure]
R TE
V a li d a ti o n
A cc u r ac y
[Figure]
A cc u r ac y
L Preliminary Results on Winograd
[Table]
rent
[Table]
misleading
[Table] [Figure]