Original: A RE BERT F AMILIES Z ERO -S HOT L EARNERS ?
Revised: A RE BERT F AMILIES Z ERO -S HOT L EARNERS ?

Original: A
Revised: A

Original: S TUDY ON T HEIR P OTENTIAL AND L IMITATIONS
Revised: S TUDY ON T HEIR P OTENTIAL AND L IMITATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Starting from the resurgence of deep learning, language models (LMs) have never been so popular.
Revised: Starting from the resurgence of deep learning, language models (LMs) have never been so popular.

Original: Through simply increasing model scale and data size, large LMs pre-trained with self-supervision objectives demonstrate awe-inspiring results on both task performance and generalization.
Revised: Through simply increasing model scale and data size, large LMs pre-trained with self-supervision objectives demonstrate awe-inspiring results on both task performance and generalization.

Original: At the early stage, supervised ﬁnetuning is indispensable in adapting pre-trained language models (PLMs) to downstream tasks.
Revised: At the early stage, supervised ﬁnetuning is indispensable in adapting pre-trained language models (PLMs) to downstream tasks.

Original: Later on, the sustained growth of model capacity and data size, as well as newly presented pre-training techniques, make the PLMs perform well under the few-shot setting, especially in the recent paradigm of prompt-based learning.
Revised: Later on, the sustained growth of model capacity and data size, as well as newly presented pre-training techniques, make the PLMs perform well under the few-shot setting, especially in the recent paradigm of prompt-based learning.

Original: After witnessing the success of PLMs for few-shot tasks, we propose to further study the potential and limitations of PLMs for the zero-shot setting.
Revised: After witnessing the success of PLMs for few-shot tasks, we propose to further study the potential and limitations of PLMs for the zero-shot setting.

Original: We utilize 3 models from the most popular BERT family to launch the empirical study on 20 different datasets.
Revised: We utilize 3 models from the most popular BERT family to launch the empirical study on 20 different datasets.

Original: We are surprised to ﬁnd that a simple Multi-Null Prompting (without manually/automatically created prompts) strategy can yield very promising results on a few widely-used datasets, e.g., 86 .
Revised: We are surprised to ﬁnd that a simple Multi-Null Prompting (without manually/automatically created prompts) strategy can yield very promising results on a few widely-used datasets, e.g., 86 .

Original: 59%( ± 0 . 59) accuracy on the
Revised: 59%( ± 0 . 59) accuracy on the

Original: IMDB dataset, and 86 .
Revised: IMDB dataset, and 86 .

Original: 22%( ± 2 . 71) accuracy on the Amazon dataset, which outperforms manually created prompts without engineering in achieving much better and stable performance with the accuracy of 74 .
Revised: 22%( ± 2 . 71) accuracy on the Amazon dataset, which outperforms manually created prompts without engineering in achieving much better and stable performance with the accuracy of 74 .

Original: 06%( ± 13 . 04) , 75 .
Revised: 06%( ± 13 . 04) , 75 .

Original: 54%( ± 11 . 77) for comparison.
Revised: 54%( ± 11 . 77) for comparison.

Original: However, we also observe some limitations of PLMs under the zero-shot setting, particularly for the language understanding tasks (e.g., GLUE).
Revised: However, we also observe some limitations of PLMs under the zero-shot setting, particularly for the language understanding tasks (e.g., GLUE).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, the Natural Language Processing (NLP) community have witnessed the explosive development of pre-trained language models (PLMs) such as GPT series (Radford et al.
Revised: In recent years, the Natural Language Processing (NLP) community have witnessed the explosive development of pre-trained language models (PLMs) such as GPT series (Radford et al.

Original: , a;b; Brown et al., 2020), BERT family, represented by (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019), and encoder-decoder models like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), and marvelled at their impressive task performance and generalization ability.
Revised: , a;b; Brown et al., 2020), BERT family, represented by (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019), and encoder-decoder models like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), and marvelled at their impressive task performance and generalization ability.

Original: Through simple post-tuning, the rich knowledge in PLMs can be effectively transferred to various downstream tasks, i.e., the posttuning process can ﬁll the gap between pre-training objectives and task utilization.
Revised: Through simple post-tuning, the rich knowledge in PLMs can be effectively transferred to various downstream tasks, i.e., the posttuning process can ﬁll the gap between pre-training objectives and task utilization.

Original: However, the scale of recent proposed PLMs can easily spiral up to hundreds of billions and even over one trillion parameters, resulting in a dilemma that adequate post-tuning of PLMs on such a scale is difﬁcult to afford while inadequate tuning will signiﬁcantly decrease the adaptation performance.
Revised: However, the scale of recent proposed PLMs can easily spiral up to hundreds of billions and even over one trillion parameters, resulting in a dilemma that adequate post-tuning of PLMs on such a scale is difﬁcult to afford while inadequate tuning will signiﬁcantly decrease the adaptation performance.

Original: To resolve the above dilemma, various prompt-based methods are introduced (Brown et al., 2020;
Revised: To resolve the above dilemma, various prompt-based methods are introduced (Brown et al., 2020;

Original: Schick & Sch¨utze, 2021a;b; Gao et al., 2021; Hambardzumyan et al., 2021) to reduce the gap between pre-training and task utilization by reformulating the downstream tasks to be more like the pre-training process.
Revised: Schick & Sch¨utze, 2021a;b; Gao et al., 2021; Hambardzumyan et al., 2021) to reduce the gap between pre-training and task utilization by reformulating the downstream tasks to be more like the pre-training process.

Original: With a more uniﬁed task formulation, the post-tuning efforts are signiﬁcantly reduced.
Revised: With a more uniﬁed task formulation, the post-tuning efforts are signiﬁcantly reduced.

Original: Take a simple text classiﬁcation case for example.
Revised: Take a simple text classiﬁcation case for example.

Original: Given a sentence “This food is delicious!”
Revised: Given a sentence “This food is delicious!”

Original: as input, conventional methods based on PLMs require supervised post-tuning to ﬁll the gap between masked language model (MLM) objective (i.e., predicting the masked word of “This food is [MASK]!” ) and the expected classiﬁcation output of 0 , 1 , . . . , n .
Revised: as input, conventional methods based on PLMs require supervised post-tuning to ﬁll the gap between masked language model (MLM) objective (i.e., predicting the masked word of “This food is [MASK]!” ) and the expected classiﬁcation output of 0 , 1 , . . . , n .

Original: Alternatively, prompt-based learning transforms the input text into the MLM task by appending a prompt, e.g., “This food is delicious!
Revised: Alternatively, prompt-based learning transforms the input text into the MLM task by appending a prompt, e.g., “This food is delicious!

Original: The sentiment of this sentence is [MASK]” , to make PLMs perform prediction directly, where the expected classiﬁcation outputs are extracted from the output of the masked position, e.g., re-scaling the probabilities of a pre-deﬁned label word pair “positive, negative” .
Revised: The sentiment of this sentence is [MASK]” , to make PLMs perform prediction directly, where the expected classiﬁcation outputs are extracted from the output of the masked position, e.g., re-scaling the probabilities of a pre-deﬁned label word pair “positive, negative” .

Original: In doing so, the efforts of post-tuning PLMs and training new modules for downstream tasks are no longer needed but requiring selective prompts and effective strategy to extract task output from the MLM prediction.
Revised: In doing so, the efforts of post-tuning PLMs and training new modules for downstream tasks are no longer needed but requiring selective prompts and effective strategy to extract task output from the MLM prediction.

Original: Through exploring three different paradigms of prompt designing strategies ( hand-crafted, discrete prompt, continuous prompt ) and various answer engineering (e.g., selecting label words for the classiﬁcation tasks), prompt-based methods have achieved very impressive and appealing results on the BERT family, especially for the few-shot setting (Schick & Sch¨utze, 2021a;b; Gao et al., 2021;
Revised: Through exploring three different paradigms of prompt designing strategies ( hand-crafted, discrete prompt, continuous prompt ) and various answer engineering (e.g., selecting label words for the classiﬁcation tasks), prompt-based methods have achieved very impressive and appealing results on the BERT family, especially for the few-shot setting (Schick & Sch¨utze, 2021a;b; Gao et al., 2021;

Original: Liu et al., 2021b; Perez et al., 2021).
Revised: Liu et al., 2021b; Perez et al., 2021).

Original: With the success of prompting methods in the few-shot setting, it is natural to explore their effectiveness in a more challenge setting, i.e., the zero-shot scenario.
Revised: With the success of prompting methods in the few-shot setting, it is natural to explore their effectiveness in a more challenge setting, i.e., the zero-shot scenario.

Original: Our Contributions.
Revised: Our Contributions.

Original: We (1) launch a thorough empirical study to test the zero-shot capabilities of representative PLMs from BERT family; (2) propose a few simple yet effective strategies to improve the zero-shot performance and robustness (3) along with coarse-to-ﬁne studies to analyze and understand the effect of each strategy; (4) observe a few essential limitations and give the possible reasons, which might shed light on the future work of zero-shot prompting on the BERT family.
Revised: We (1) launch a thorough empirical study to test the zero-shot capabilities of representative PLMs from BERT family; (2) propose a few simple yet effective strategies to improve the zero-shot performance and robustness (3) along with coarse-to-ﬁne studies to analyze and understand the effect of each strategy; (4) observe a few essential limitations and give the possible reasons, which might shed light on the future work of zero-shot prompting on the BERT family.

Original: Most Surprising Findings.
Revised: Most Surprising Findings.

Original: Though the designs of experiments are simple and straightforward, we still observe a few surprising ﬁndings.
Revised: Though the designs of experiments are simple and straightforward, we still observe a few surprising ﬁndings.

Original: First, Multi-Null Prompt, a simple modiﬁcation of the
Revised: First, Multi-Null Prompt, a simple modiﬁcation of the

Original: Null Prompt method by concatenating multiple [MASK] tokens, can outperform manual prompts in text classiﬁcation tasks.
Revised: Null Prompt method by concatenating multiple [MASK] tokens, can outperform manual prompts in text classiﬁcation tasks.

Original: More surprisingly, without using [MASK] tokens, inserting multiple random tokens even achieve improved performance.
Revised: More surprisingly, without using [MASK] tokens, inserting multiple random tokens even achieve improved performance.

Original: Besides, even breaking the form of natural language, inserting multiple [MASK] tokens at the random position of the text also performs well.
Revised: Besides, even breaking the form of natural language, inserting multiple [MASK] tokens at the random position of the text also performs well.

Original: Paper Structure.
Revised: Paper Structure.

Original: The rest of this paper is organized as follows: We present some preliminaries about the utilization of PLMs in Section 2, and we test some basic models adapted from the few-shot setting and propose two new strategies for zero-shot setting in Section 3, following with a coarseto-ﬁne study in Section 4.
Revised: The rest of this paper is organized as follows: We present some preliminaries about the utilization of PLMs in Section 2, and we test some basic models adapted from the few-shot setting and propose two new strategies for zero-shot setting in Section 3, following with a coarseto-ﬁne study in Section 4.

Original: Section 5 and 6 present the possible limitations of the BERT family.
Revised: Section 5 and 6 present the possible limitations of the BERT family.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: To study and calibrate the effect of prompt-based methods for the zero-shot setting, we ﬁrst present two typical paradigms that perform well for supervised post-tuning and the few-shot setting, including ﬁne-tuning methods and prompt-based learning.
Revised: To study and calibrate the effect of prompt-based methods for the zero-shot setting, we ﬁrst present two typical paradigms that perform well for supervised post-tuning and the few-shot setting, including ﬁne-tuning methods and prompt-based learning.

Original: These methods will be extended to the zero-shot scenario in Section 3 to calibrate the potential of prompting augmented with simple strategies.
Revised: These methods will be extended to the zero-shot scenario in Section 3 to calibrate the potential of prompting augmented with simple strategies.

Original: 2. F INE -T UNING OF PLM S
Revised: 2. F INE -T UNING OF PLM S

Original: Starting from the occurrence of large-scale PLMs, ﬁne-tuning methods have almost been indispensable and are still strong competitors in recent papers to mitigate the gap between the pre-training objectives and downstream tasks (Devlin et al., 2019; Liu et al., 2019;
Revised: Starting from the occurrence of large-scale PLMs, ﬁne-tuning methods have almost been indispensable and are still strong competitors in recent papers to mitigate the gap between the pre-training objectives and downstream tasks (Devlin et al., 2019; Liu et al., 2019;

Original: Lan et al., 2019; Radford et al.
Revised: Lan et al., 2019; Radford et al.

Original: , a;b; Brown et al., 2020; Raffel et al., 2020).
Revised: , a;b; Brown et al., 2020; Raffel et al., 2020).

Original: For text classiﬁcation and natural language understanding tasks, PLMs are mainly utilized to obtain the contextualized representation of text.
Revised: For text classiﬁcation and natural language understanding tasks, PLMs are mainly utilized to obtain the contextualized representation of text.

Original: Each given text input x = ( x 1 , x 2 ,...)
Revised: Each given text input x = ( x 1 , x 2 ,...)

Original: ﬁrst concatenates a [ CLS ] token and a [ SEP ] token at the beginning and the end of x , respectively.
Revised: ﬁrst concatenates a [ CLS ] token and a [ SEP ] token at the beginning and the end of x , respectively.

Original: Then, PLMs process the input text ([ CLS ] , x, [ SEP ]) to obtain the corresponded vector representations h
Revised: Then, PLMs process the input text ([ CLS ] , x, [ SEP ]) to obtain the corresponded vector representations h

Original: = ( h [ CLS ] , h 1 , h 2 ,..., h [ SEP ] ).
Revised: = ( h [ CLS ] , h 1 , h 2 ,..., h [ SEP ] ).

Original: In most cases, either h [ CLS ] or the output of mean pooling over each element of h is utilized as the aggregated representation of input text x , denoted as h x , which is further processed by task-speciﬁc modules to obtain the expected output label y .
Revised: In most cases, either h [ CLS ] or the output of mean pooling over each element of h is utilized as the aggregated representation of input text x , denoted as h x , which is further processed by task-speciﬁc modules to obtain the expected output label y .

Original: The task-speciﬁc modules are usually implemented as a stack of multiple linear layers (MLPs) along with a softmax classiﬁer to compute the output labels, written by
Revised: The task-speciﬁc modules are usually implemented as a stack of multiple linear layers (MLPs) along with a softmax classiﬁer to compute the output labels, written by

Original: [Equation]
Revised: [Equation]

Original: where W refers to trainable parameters of the classiﬁer, which are trained from scratch.
Revised: where W refers to trainable parameters of the classiﬁer, which are trained from scratch.

Original: 2. P ROMPT -B ASED M ETHODS
Revised: 2. P ROMPT -B ASED M ETHODS

Original: Instead of performing ﬁne-tuning to adapt PLMs to different downstream tasks, prompt-based learning leverages prompt to reformulate the downstream tasks to be more like the pre-training task, which is usually a language modeling task.
Revised: Instead of performing ﬁne-tuning to adapt PLMs to different downstream tasks, prompt-based learning leverages prompt to reformulate the downstream tasks to be more like the pre-training task, which is usually a language modeling task.

Original: Formally, the input text from the downstream task is ﬁrst mapped to a prompt by a prompting function f prompt ( · ) which comprises two stages:
Revised: Formally, the input text from the downstream task is ﬁrst mapped to a prompt by a prompting function f prompt ( · ) which comprises two stages:

Original: • A template is ﬁrst constructed and applied to combine prompt tokens, input text slot [X], and answer slot
Revised: • A template is ﬁrst constructed and applied to combine prompt tokens, input text slot [X], and answer slot

Original: [Y].
Revised: [Y].

Original: For instance, “[X] The sentiment of this sentence is [Y]” is a feasible template for the sentiment classiﬁcation task, where [Y] is [MASK] for the BERT family.
Revised: For instance, “[X] The sentiment of this sentence is [Y]” is a feasible template for the sentiment classiﬁcation task, where [Y] is [MASK] for the BERT family.

Original: • Then, f prompt ( · ) ﬁlls the slot [X] with the input text, e.g., converting the above-mentioned template to “This food is delicious!
Revised: • Then, f prompt ( · ) ﬁlls the slot [X] with the input text, e.g., converting the above-mentioned template to “This food is delicious!

Original: The sentiment of this sentence is [Y]” .
Revised: The sentiment of this sentence is [Y]” .

Original: After mapping the input text to prompt, the model predicts the token in the position of answer slot [Y], which resembles predicting possible tokens during the pre-training stage.
Revised: After mapping the input text to prompt, the model predicts the token in the position of answer slot [Y], which resembles predicting possible tokens during the pre-training stage.

Original: Different from predicting possible tokens in the whole vocabulary in pre-training, prompt-based learning only calculates the possibility of the pre-set tokens, called label words.
Revised: Different from predicting possible tokens in the whole vocabulary in pre-training, prompt-based learning only calculates the possibility of the pre-set tokens, called label words.

Original: Take the aforementioned prompt template for example.
Revised: Take the aforementioned prompt template for example.

Original: The pre-set tokens can be “positive” and “negative” .
Revised: The pre-set tokens can be “positive” and “negative” .

Original: Then, a Verbalizer (Liu et al., 2021a) maps the highest-scoring label word (e.g., “positive” for “This food is delicious!
Revised: Then, a Verbalizer (Liu et al., 2021a) maps the highest-scoring label word (e.g., “positive” for “This food is delicious!

Original: The sentiment of this sentence is [Y]” ) to its corresponding label indices { 0 , 1 , . . . , n } as the output.
Revised: The sentiment of this sentence is [Y]” ) to its corresponding label indices { 0 , 1 , . . . , n } as the output.

Original:  O N THE P OTENTIAL OF P ROMPTING FOR Z ERO -S HOT S ETTING
Revised:  O N THE P OTENTIAL OF P ROMPTING FOR Z ERO -S HOT S ETTING

Original: In this section, we ﬁrst present a few basic models adopted from existing works and their implementation details.
Revised: In this section, we ﬁrst present a few basic models adopted from existing works and their implementation details.

Original: We then test the performance of these basic models on four widely utilized datasets under the zero-shot setting.
Revised: We then test the performance of these basic models on four widely utilized datasets under the zero-shot setting.

Original: After calibrating the zero-shot performance of existing methods from other settings, we introduce two simple strategies upon the basic models for further exploration.
Revised: After calibrating the zero-shot performance of existing methods from other settings, we introduce two simple strategies upon the basic models for further exploration.

Original: 3. S ETTINGS
Revised: 3. S ETTINGS

Original: Basic Models.
Revised: Basic Models.

Original: In the ﬁne-tuning of PLMs, task-speciﬁc models mainly comprise two essential components, i.e., PLMs for mapping input text to the contextualized representation and the taskspeciﬁc classiﬁer learned from scratch.
Revised: In the ﬁne-tuning of PLMs, task-speciﬁc models mainly comprise two essential components, i.e., PLMs for mapping input text to the contextualized representation and the taskspeciﬁc classiﬁer learned from scratch.

Original: In the prompt-based learning, the input text is represented by the output hidden state of PLMs at the position of the answer slot [Y], and the task classiﬁer is part of the PLMs, i.e., the MLM head, which is utilized to compute the probabilities of the pre-set label words.
Revised: In the prompt-based learning, the input text is represented by the output hidden state of PLMs at the position of the answer slot [Y], and the task classiﬁer is part of the PLMs, i.e., the MLM head, which is utilized to compute the probabilities of the pre-set label words.

Original: In the zero-shot setting, it is inapplicable to train a task-speciﬁc classiﬁer, and thus the MLM head classiﬁer utilized in prompt-based learning is a natural and feasible alternative.
Revised: In the zero-shot setting, it is inapplicable to train a task-speciﬁc classiﬁer, and thus the MLM head classiﬁer utilized in prompt-based learning is a natural and feasible alternative.

Original: To this end, we create three basic models based on different representation aggregation strategies in conventional methods of ﬁne-tuning PLMs, including
Revised: To this end, we create three basic models based on different representation aggregation strategies in conventional methods of ﬁne-tuning PLMs, including

Original: • CLS.
Revised: • CLS.

Original: The output hidden state of the [CLS] token, i
Revised: The output hidden state of the [CLS] token, i

Original: e., the ﬁrst token before the input text, is utilized as the aggregated representation of the input text.
Revised: e., the ﬁrst token before the input text, is utilized as the aggregated representation of the input text.

Original: • SEP.
Revised: • SEP.

Original: The output hidden state of the [SEP] token, i.e., the token appended to the end of the input text, is used as the aggregated representation of the input text.
Revised: The output hidden state of the [SEP] token, i.e., the token appended to the end of the input text, is used as the aggregated representation of the input text.

Original: • Mean Pooling.
Revised: • Mean Pooling.

Original: The hidden states of all tokens in the input text (without [CLS] and [SEP]) are averaged as the representation of the input text.
Revised: The hidden states of all tokens in the input text (without [CLS] and [SEP]) are averaged as the representation of the input text.

Original: The aggregated representations of the input text are then processed by the MLM head classiﬁer to predict task-speciﬁc labels.
Revised: The aggregated representations of the input text are then processed by the MLM head classiﬁer to predict task-speciﬁc labels.

Original: Note that we do not utilize prompt to process the input text in the above three basic models and only borrow the MLM head classiﬁer and the pre-set label words from the prompt-based learning to replace the task-speciﬁc output layer that requires supervised training.
Revised: Note that we do not utilize prompt to process the input text in the above three basic models and only borrow the MLM head classiﬁer and the pre-set label words from the prompt-based learning to replace the task-speciﬁc output layer that requires supervised training.

Original: As for the prompt-based learning, we also extend three representative few-shot methods from existing literature and implement the following methods in zero-shot prompt-based learning.
Revised: As for the prompt-based learning, we also extend three representative few-shot methods from existing literature and implement the following methods in zero-shot prompt-based learning.

Original: • Manual Prompt (Prior).
Revised: • Manual Prompt (Prior).

Original: We use the handcrafted prompts designed by Schick & Sch¨utze (2021a;b);
Revised: We use the handcrafted prompts designed by Schick & Sch¨utze (2021a;b);

Original: Hu et al.
Revised: Hu et al.

Original: (2021), which are tuned using large validation sets or selected by validation data (Logan IV et al., 2021).
Revised: (2021), which are tuned using large validation sets or selected by validation data (Logan IV et al., 2021).

Original: The utilized prompts are given in Appendix A.1.
Revised: The utilized prompts are given in Appendix A.1.

Original: • Manual Prompt # .
Revised: • Manual Prompt # .

Original: We also manually design a few intuitive prompts that are not selected carefully using validation sets to study the inﬂuence of prompt engineering for the zero-shot setting, where the manually designed prompts are presented in the Appendix A.1.
Revised: We also manually design a few intuitive prompts that are not selected carefully using validation sets to study the inﬂuence of prompt engineering for the zero-shot setting, where the manually designed prompts are presented in the Appendix A.1.

Original: • Null Prompt.
Revised: • Null Prompt.

Original: Following Logan IV et al.
Revised: Following Logan IV et al.

Original: (2021) to simplify prompt engineering, we only insert a [MASK] token at the end of the text, i.e., using the prompt template “[X][MASK]” .
Revised: (2021) to simplify prompt engineering, we only insert a [MASK] token at the end of the text, i.e., using the prompt template “[X][MASK]” .

Original: We also introduce
Revised: We also introduce

Original: three models that
Revised: three models that

Original: use unlabeled data or external knowledge
Revised: use unlabeled data or external knowledge

Original: from
Revised: from

Original: existing works
Revised: existing works

Original: [Table]
Revised: [Table]

Original: • NSP-BERT.
Revised: • NSP-BERT.

Original: Sun et al.
Revised: Sun et al.

Original: (2021) reformulate text classiﬁcation task into text entailment-style tasks and then use the Next Sentence Prediction (NSP) head to predict the result.
Revised: (2021) reformulate text classiﬁcation task into text entailment-style tasks and then use the Next Sentence Prediction (NSP) head to predict the result.

Original: The templates we used are shown in Appendix A.1.
Revised: The templates we used are shown in Appendix A.1.

Original: • LOTClass.
Revised: • LOTClass.

Original: Meng et al.
Revised: Meng et al.

Original: (2020) use unlabeled data to ﬁnd the words similar to label names and introduce a self-training approach to induce a classiﬁer.
Revised: (2020) use unlabeled data to ﬁnd the words similar to label names and introduce a self-training approach to induce a classiﬁer.

Original: • KPT.
Revised: • KPT.

Original: Hu et al.
Revised: Hu et al.

Original: (2021) propose a prompt-based method that uses external knowledge to expand the space of label name.
Revised: (2021) propose a prompt-based method that uses external knowledge to expand the space of label name.

Original: Datasets.
Revised: Datasets.

Original: We carry out experiments on several widely-acknowledged datasets targeting different aspects of text classiﬁcation to study the performance of the above-described methods and evaluate newly proposed strategies that are designed for exploring the potential of zero-shot setting.
Revised: We carry out experiments on several widely-acknowledged datasets targeting different aspects of text classiﬁcation to study the performance of the above-described methods and evaluate newly proposed strategies that are designed for exploring the potential of zero-shot setting.

Original: Speciﬁcally, we follow Meng et al.
Revised: Speciﬁcally, we follow Meng et al.

Original: and systematically evaluate these methods on 2 topic classiﬁcation tasks and 2 sentiment classiﬁcation tasks, which are IMDB (Maas et al., 2011), Amazon (McAuley & Leskovec, 2013), AG News (Zhang et al., 2015), and DBPedia (Lehmann et al., 2015).
Revised: and systematically evaluate these methods on 2 topic classiﬁcation tasks and 2 sentiment classiﬁcation tasks, which are IMDB (Maas et al., 2011), Amazon (McAuley & Leskovec, 2013), AG News (Zhang et al., 2015), and DBPedia (Lehmann et al., 2015).

Original: For each task, we only leverage the test set to directly evaluate model performance without introducing validation or training sets to perform post-tuning or cherry-pick hand-crafted prompts.
Revised: For each task, we only leverage the test set to directly evaluate model performance without introducing validation or training sets to perform post-tuning or cherry-pick hand-crafted prompts.

Original: Implementation Details.
Revised: Implementation Details.

Original: We mainly conduct experiments using roberta-large as the backbone pre-trained LM.
Revised: We mainly conduct experiments using roberta-large as the backbone pre-trained LM.

Original: To conﬁrm the generalization of the proposed method, we also report the results of bert-base-uncased and albert-xxlarge-v2 , which will be discussed in the next section.
Revised: To conﬁrm the generalization of the proposed method, we also report the results of bert-base-uncased and albert-xxlarge-v2 , which will be discussed in the next section.

Original: Our model implementation is based the open toolkit Huggingface Transformers (Wolf et al., 2020) 1 .
Revised: Our model implementation is based the open toolkit Huggingface Transformers (Wolf et al., 2020) 1 .

Original: To load the data of different text classiﬁcation tasks, we also use the open toolkit Huggingface Datasets 2 (Lhoest et al., 2021).
Revised: To load the data of different text classiﬁcation tasks, we also use the open toolkit Huggingface Datasets 2 (Lhoest et al., 2021).

Original: More accurately, we use imdb , amazon polarity , ag news , dbpedia 14 .
Revised: More accurately, we use imdb , amazon polarity , ag news , dbpedia 14 .

Original: 3. R ESULTS AND E XPLORATIONS
Revised: 3. R ESULTS AND E XPLORATIONS

Original: Overall Performance of Basic Models.
Revised: Overall Performance of Basic Models.

Original: As shown in Table 1, the combination of simple representation aggregation methods and MLM head classiﬁer along with the pre-set label words can only achieve a preliminary performance, represented by the low accuracy of CLS and SEP models (34.72% and 44.14% on average).
Revised: As shown in Table 1, the combination of simple representation aggregation methods and MLM head classiﬁer along with the pre-set label words can only achieve a preliminary performance, represented by the low accuracy of CLS and SEP models (34.72% and 44.14% on average).

Original: Through utilizing all tokens in the input text, Mean Pooling performs much better and is stable than the previous two methods, resulting in an accuracy of 57.68% on average.
Revised: Through utilizing all tokens in the input text, Mean Pooling performs much better and is stable than the previous two methods, resulting in an accuracy of 57.68% on average.

Original: With simpliﬁed human efforts for designing prompts, the Manual Prompt # signiﬁcantly improves the performance.
Revised: With simpliﬁed human efforts for designing prompts, the Manual Prompt # signiﬁcantly improves the performance.

Original: The cherry-picked manual prompts further increase the power of the Manual Prompt method.
Revised: The cherry-picked manual prompts further increase the power of the Manual Prompt method.

Original: Without human efforts for designing prompt, the Null Prompt method (only with an extra [MASK] as the prompt token) yields nearly comparable results with Manual
Revised: Without human efforts for designing prompt, the Null Prompt method (only with an extra [MASK] as the prompt token) yields nearly comparable results with Manual

Original: [Table]
Revised: [Table]

Original: Prompt # but outperforms these non-prompting zero-shot methods.
Revised: Prompt # but outperforms these non-prompting zero-shot methods.

Original: Through leveraging extra knowledge and unlabelled data, LOTClass and KPT achieve the best results, which are almost comparable with SOTA few-shot learning methods and even supervised learning models.
Revised: Through leveraging extra knowledge and unlabelled data, LOTClass and KPT achieve the best results, which are almost comparable with SOTA few-shot learning methods and even supervised learning models.

Original: In short, unlabeled data and extra knowledge are most effective for the zero-shot setting, and human efforts take second place.
Revised: In short, unlabeled data and extra knowledge are most effective for the zero-shot setting, and human efforts take second place.

Original: Prompt-based learning (even the Null Prompt setting) is more effective than tuning-based zero-shot methods.
Revised: Prompt-based learning (even the Null Prompt setting) is more effective than tuning-based zero-shot methods.

Original: However, neither much human effort nor unlabelled data, nor extra knowledge is ideal for realizing the full potential of PLMs in generalizing to different downstream tasks.
Revised: However, neither much human effort nor unlabelled data, nor extra knowledge is ideal for realizing the full potential of PLMs in generalizing to different downstream tasks.

Original: Thus, we will focus on the Null Prompt setting in the rest of this section to explore the potential of PLMs.
Revised: Thus, we will focus on the Null Prompt setting in the rest of this section to explore the potential of PLMs.

Original: Recall that, in our challenging setting, we cannot utilize any external information other than label name, which is necessary to tell the model to classify the text from which aspect.
Revised: Recall that, in our challenging setting, we cannot utilize any external information other than label name, which is necessary to tell the model to classify the text from which aspect.

Original: To fully use the
Revised: To fully use the

Original: PLMs knowledge, we propose two simple yet effective strategies at different inference stages, which can be utilized simultaneously.
Revised: PLMs knowledge, we propose two simple yet effective strategies at different inference stages, which can be utilized simultaneously.

Original: Their effectiveness is studied in the following paragraphs.
Revised: Their effectiveness is studied in the following paragraphs.

Original: Ensemble of Multiple-Null Prompt Masks.
Revised: Ensemble of Multiple-Null Prompt Masks.

Original: We ﬁrst propose to enhance the capability of the
Revised: We ﬁrst propose to enhance the capability of the

Original: Null Prompt method with multiple prompt [MASK] tokens 3 , named Multi Null Prompt, i.e., inserting multiple prompt [MASK] at different positions of the input text (e.g., the end, the head, the middle).
Revised: Null Prompt method with multiple prompt [MASK] tokens 3 , named Multi Null Prompt, i.e., inserting multiple prompt [MASK] at different positions of the input text (e.g., the end, the head, the middle).

Original: The multiple answers from the prompt masks are aggregated by different ensemble methods.
Revised: The multiple answers from the prompt masks are aggregated by different ensemble methods.

Original: The positions of prompt [MASK] and ensemble strategies will be thoroughly studied in
Revised: The positions of prompt [MASK] and ensemble strategies will be thoroughly studied in

Original: Section 4.
Revised: Section 4.

Original: Table 1 presents the results of inserting one single [MASK] at both the beginning and the end of the input text, i.e., in the form of “[MASK][X][MASK]” .
Revised: Table 1 presents the results of inserting one single [MASK] at both the beginning and the end of the input text, i.e., in the form of “[MASK][X][MASK]” .

Original: We can see that the simple Multi
Revised: We can see that the simple Multi

Original: Null Prompt method results in substantial performance gains over the Null Prompt basic model.
Revised: Null Prompt method results in substantial performance gains over the Null Prompt basic model.

Original: Searching for More Label Words.
Revised: Searching for More Label Words.

Original: This strategy directly utilizes cosine distance as the metric to search and pick up the words with similar embeddings to the label names.
Revised: This strategy directly utilizes cosine distance as the metric to search and pick up the words with similar embeddings to the label names.

Original: It can be easily seen from
Revised: It can be easily seen from

Original: Table 1 that with the enhancement of more label words, the Null Prompt method gains 9.05% on average, which is almost comparable with the Manual Prompt (Prior) that needs considerable human efforts.
Revised: Table 1 that with the enhancement of more label words, the Null Prompt method gains 9.05% on average, which is almost comparable with the Manual Prompt (Prior) that needs considerable human efforts.

Original: Experimental result on the Multi-Null Prompt model also shows signiﬁcant improvement, which even performs better than Manual Prompt with non-trivial human efforts.
Revised: Experimental result on the Multi-Null Prompt model also shows signiﬁcant improvement, which even performs better than Manual Prompt with non-trivial human efforts.

Original: We can conclude from these experimental results that simple strategies without human efforts or extra knowledge or unlabelled data can also release the potential of PLMs and signiﬁcantly enhance the zero-shot performance.
Revised: We can conclude from these experimental results that simple strategies without human efforts or extra knowledge or unlabelled data can also release the potential of PLMs and signiﬁcantly enhance the zero-shot performance.

Original: It is possible to design more general and effective strategies to facilitate the zero-shot utilization of PLMs, even for the BERT family.
Revised: It is possible to design more general and effective strategies to facilitate the zero-shot utilization of PLMs, even for the BERT family.

Original:  C OARSE - TO -F INE S TUDY OF P ROMPTING S TRATEGIES
Revised:  C OARSE - TO -F INE S TUDY OF P ROMPTING S TRATEGIES

Original: In this section, we conduct coarse-to-ﬁne studies to analyze the reasons behind the impressive performance of the simple Multi-Null Prompt method, including the position of prompt [MASK], the number of prompt [MASK], the possible alternative of prompt [MASK], the effect of similar label words, and the inﬂuence of different ensemble methods for multiple predicted answers.
Revised: In this section, we conduct coarse-to-ﬁne studies to analyze the reasons behind the impressive performance of the simple Multi-Null Prompt method, including the position of prompt [MASK], the number of prompt [MASK], the possible alternative of prompt [MASK], the effect of similar label words, and the inﬂuence of different ensemble methods for multiple predicted answers.

Original: We also conduct experiments on different backbone PLMs to test the generalization of the proposed method.
Revised: We also conduct experiments on different backbone PLMs to test the generalization of the proposed method.

Original: 4. T HE P OSITION AND N UMBER OF P ROMPT [MASK]
Revised: 4. T HE P OSITION AND N UMBER OF P ROMPT [MASK]

Original: We ﬁrst study the effect of the position and number of the prompt [MASK].
Revised: We ﬁrst study the effect of the position and number of the prompt [MASK].

Original: Figure 1 shows that inserting [MASK] tokens at both the start and the end of the text outperforms inserting at the start or end separately.
Revised: Figure 1 shows that inserting [MASK] tokens at both the start and the end of the text outperforms inserting at the start or end separately.

Original: Since we do not utilize manual prompts, we do not need to follow prior promptbased methods to insert prompts without breaking the form of natural language.
Revised: Since we do not utilize manual prompts, we do not need to follow prior promptbased methods to insert prompts without breaking the form of natural language.

Original: Therefore, we insert the [MASK] tokens at the random position of the text.
Revised: Therefore, we insert the [MASK] tokens at the random position of the text.

Original: The results are reported in Table 2, which shows that even though breaking the form of natural language, it can still achieve promising performance, and the performance improves as the number of [MASK] tokens increases.
Revised: The results are reported in Table 2, which shows that even though breaking the form of natural language, it can still achieve promising performance, and the performance improves as the number of [MASK] tokens increases.

Original: We also study the changes in the performance when the number of [MASK] tokens differs.
Revised: We also study the changes in the performance when the number of [MASK] tokens differs.

Original: Figure 1 shows that Multi-Null Prompt is robust to the number of [MASK]
Revised: Figure 1 shows that Multi-Null Prompt is robust to the number of [MASK]

Original: tokens
Revised: tokens

Original: [Table]
Revised: [Table]

Original: 4. T HE P OSSIBLE A LTERNATIVE OF P ROMPT [MASK]
Revised: 4. T HE P OSSIBLE A LTERNATIVE OF P ROMPT [MASK]

Original: [Table]
Revised: [Table]

Original: We also explore the question that Must we use [MASK] token as answer slot
Revised: We also explore the question that Must we use [MASK] token as answer slot

Original: [Y]?
Revised: [Y]?

Original: The results are reported in Table 3.
Revised: The results are reported in Table 3.

Original: In this experiment, we insert one token as answer slot [Y] at both the start and end of the text and utilize 50 correlated label words.
Revised: In this experiment, we insert one token as answer slot [Y] at both the start and end of the text and utilize 50 correlated label words.

Original: The results show that even though we do not use [MASK] token as answer slot [Y], Multi-Null Prompt still works and achieves even better performance.
Revised: The results show that even though we do not use [MASK] token as answer slot [Y], Multi-Null Prompt still works and achieves even better performance.

Original: This is surprising because all the prior prompt-based methods use [MASK] token as answer slot
Revised: This is surprising because all the prior prompt-based methods use [MASK] token as answer slot

Original: [Y].
Revised: [Y].

Original: We observe that compared to other tokens, the token ‘.’ achieve the worst performance.
Revised: We observe that compared to other tokens, the token ‘.’ achieve the worst performance.

Original: We speculate that this phenomenon is caused by the intrinsic attribute of MLM objectives.
Revised: We speculate that this phenomenon is caused by the intrinsic attribute of MLM objectives.

Original: MLM task not only substitutes tokens with prompt [MASK]s but also with random tokens, which make the BERT family have the ability to recognize the mistaken tokens.
Revised: MLM task not only substitutes tokens with prompt [MASK]s but also with random tokens, which make the BERT family have the ability to recognize the mistaken tokens.

Original: Hence, if we insert random tokens, the model can recognize inserted tokens easily and represent them as [MASK] tokens, while the model may treat the token as a reasonable element of the whole text rather than the prompt [MASK] if we insert token like ‘.’
Revised: Hence, if we insert random tokens, the model can recognize inserted tokens easily and represent them as [MASK] tokens, while the model may treat the token as a reasonable element of the whole text rather than the prompt [MASK] if we insert token like ‘.’

Original: . 4. T HE E FFECT OF D IFFERENT E NSEMBLE M ETHODS FOR M ULTIPLE P ROMPT A NSWERS
Revised: . 4. T HE E FFECT OF D IFFERENT E NSEMBLE M ETHODS FOR M ULTIPLE P ROMPT A NSWERS

Original: To utilize multiple prompt answers from different prompt [MASK]s, we implement four different ensemble methods, including 1) Single Position.
Revised: To utilize multiple prompt answers from different prompt [MASK]s, we implement four different ensemble methods, including 1) Single Position.

Original: Only one of the prompt [MASK] positions is
Revised: Only one of the prompt [MASK] positions is

Original: [Table]
Revised: [Table]

Original: utilized to obtain the prompt answer, which is then processed to output the task-speciﬁc label; 2)
Revised: utilized to obtain the prompt answer, which is then processed to output the task-speciﬁc label; 2)

Original: Mean Pooling.
Revised: Mean Pooling.

Original: The output logits at all prompt [MASK] positions are averaged as the ensemble output; 3) Max Logits.
Revised: The output logits at all prompt [MASK] positions are averaged as the ensemble output; 3) Max Logits.

Original: The prompt [MASK] position with the maximum probabilities of the preset label words (i.e., the sum of all the probabilities of each label word) is selected for computing the task-speciﬁc label; 4) Majority.
Revised: The prompt [MASK] position with the maximum probabilities of the preset label words (i.e., the sum of all the probabilities of each label word) is selected for computing the task-speciﬁc label; 4) Majority.

Original: The majority voting result of different prompt answers is utilized as the ﬁnal output.
Revised: The majority voting result of different prompt answers is utilized as the ﬁnal output.

Original: As presented in Figure 1, the mean pooling method performs best in aggregating the outputs from multiple prompt [MASK]s, and the majority method takes second place.
Revised: As presented in Figure 1, the mean pooling method performs best in aggregating the outputs from multiple prompt [MASK]s, and the majority method takes second place.

Original: 4. T HE E FFECT OF S IMILAR L ABEL W ORDS
Revised: 4. T HE E FFECT OF S IMILAR L ABEL W ORDS

Original: In this experiment, we study the effect of the number of similar label words, and the results are shown in Figure 2.
Revised: In this experiment, we study the effect of the number of similar label words, and the results are shown in Figure 2.

Original: From the ﬁgure, we can conclude that the performance does not change so much when the number of similar label words varies, which proves that searching for more label words by the cosine similarity of embeddings is a simple but effective and robust method to improve the performance.
Revised: From the ﬁgure, we can conclude that the performance does not change so much when the number of similar label words varies, which proves that searching for more label words by the cosine similarity of embeddings is a simple but effective and robust method to improve the performance.

Original: 4. M ODEL V ARIANTS
Revised: 4. M ODEL V ARIANTS

Original: To further conﬁrm the effectiveness of Multi-Null
Revised: To further conﬁrm the effectiveness of Multi-Null

Original: Prompt, we also use BERT and ALBERT to conduct experiments on IMDB, Amazon, AG News, DBPedia.
Revised: Prompt, we also use BERT and ALBERT to conduct experiments on IMDB, Amazon, AG News, DBPedia.

Original: The results are reported in Table 4.
Revised: The results are reported in Table 4.

Original: The signiﬁcant improvement on different PLMs shows that the beneﬁts of Multi-Null Prompt are brought from MLM tasks and can be applied to BERT families.
Revised: The signiﬁcant improvement on different PLMs shows that the beneﬁts of Multi-Null Prompt are brought from MLM tasks and can be applied to BERT families.

Original:  P OSSIBLE L IMITATIONS
Revised:  P OSSIBLE L IMITATIONS

Original: To further study the prompt-based methods in zero-shot settings, we also conduct experiments on
Revised: To further study the prompt-based methods in zero-shot settings, we also conduct experiments on

Original: NLU benchmarks, GLUE, and SuperGLUE.
Revised: NLU benchmarks, GLUE, and SuperGLUE.

Original: The results are shown in Table 5 and Table 6 4 .
Revised: The results are shown in Table 5 and Table 6 4 .

Original: The results show that prompt-based methods fail in the more challenging natural language understanding tasks, though achieving promising results in zero-shot text classiﬁcation.
Revised: The results show that prompt-based methods fail in the more challenging natural language understanding tasks, though achieving promising results in zero-shot text classiﬁcation.

Original: Therefore, in this section, we analyze possible limitations of prompt-based methods in zero-shot settings by comparing results on different types of tasks and few-shot learning methods and try to give some possible reasons.
Revised: Therefore, in this section, we analyze possible limitations of prompt-based methods in zero-shot settings by comparing results on different types of tasks and few-shot learning methods and try to give some possible reasons.

Original: 5. E XPERIMENT R ESULTS OF D IFFERENT K INDS OF T ASKS
Revised: 5. E XPERIMENT R ESULTS OF D IFFERENT K INDS OF T ASKS

Original: To explore how prompt-based methods perform on different kinds of tasks, we conduct experiments on various kinds of tasks, including single-sentence tasks (SST-2, CoLA), inference tasks (MNLI, QNLI,RTE), similarity&paraphrase tasks (MRPC, QQP, STS-B, CB), question answering
Revised: To explore how prompt-based methods perform on different kinds of tasks, we conduct experiments on various kinds of tasks, including single-sentence tasks (SST-2, CoLA), inference tasks (MNLI, QNLI,RTE), similarity&paraphrase tasks (MRPC, QQP, STS-B, CB), question answering

Original: [Table]
Revised: [Table]

Original: tasks (BoolQ, MultiRC), word sense disambiguation tasks (WiC) 5 .
Revised: tasks (BoolQ, MultiRC), word sense disambiguation tasks (WiC) 5 .

Original: Table 5 and Table 6 show that manual prompts can achieve much better performance than majority class in SST-2, MNLI, QQP, comparable performance than majority class in CoLA, QNLI, RTE, BoolQ, CB, MultiRC, WiC, a worse performance than majority class in MRPC.
Revised: Table 5 and Table 6 show that manual prompts can achieve much better performance than majority class in SST-2, MNLI, QQP, comparable performance than majority class in CoLA, QNLI, RTE, BoolQ, CB, MultiRC, WiC, a worse performance than majority class in MRPC.

Original: Besides, without the help of prior knowledge in manual prompts, Multi-Null Prompt can achieve comparable results to manual prompts, except for better performance in SST-2, CoLA, and worse performance in MRPC, QQP, BoolQ.
Revised: Besides, without the help of prior knowledge in manual prompts, Multi-Null Prompt can achieve comparable results to manual prompts, except for better performance in SST-2, CoLA, and worse performance in MRPC, QQP, BoolQ.

Original: 5. T HE P OSSIBLE R EASONS
Revised: 5. T HE P OSSIBLE R EASONS

Original: For inference tasks and similarity&paraphrase tasks, we suspect the difﬁculty of designing highquality label words that can represent the meaning of the label well may cause poor performance.
Revised: For inference tasks and similarity&paraphrase tasks, we suspect the difﬁculty of designing highquality label words that can represent the meaning of the label well may cause poor performance.

Original: Since we conduct experiments in zero-shot settings, the label words are the only source for the model to achieve the task-speciﬁc information, which has a considerable impact on the performance.
Revised: Since we conduct experiments in zero-shot settings, the label words are the only source for the model to achieve the task-speciﬁc information, which has a considerable impact on the performance.

Original: The importance of label words is also emphasized in other different low resource settings, which proves our hypothesis to some extent (Meng et al., 2020; Le Scao & Rush, 2021).
Revised: The importance of label words is also emphasized in other different low resource settings, which proves our hypothesis to some extent (Meng et al., 2020; Le Scao & Rush, 2021).

Original: Meng et al.
Revised: Meng et al.

Original: (2020) leverage unsupervised data and introduce that if label names can not well explain the meaning of the label, the model can not predict well in zero-shot settings.
Revised: (2020) leverage unsupervised data and introduce that if label names can not well explain the meaning of the label, the model can not predict well in zero-shot settings.

Original: Le Scao & Rush (2021) focus on few-shot learning, and they ﬁnd that verbalizer brings more beneﬁts than prompt for small-scale data tasks.
Revised: Le Scao & Rush (2021) focus on few-shot learning, and they ﬁnd that verbalizer brings more beneﬁts than prompt for small-scale data tasks.

Original: For question answering tasks, where the prompt and label words can express the meaning of the label clearly, the failure of prompt-based methods may be caused by the scaling-up since GPT-3 also fail to achieve promising results at comparable scaling-up.
Revised: For question answering tasks, where the prompt and label words can express the meaning of the label clearly, the failure of prompt-based methods may be caused by the scaling-up since GPT-3 also fail to achieve promising results at comparable scaling-up.

Original: For word sense disambiguation tasks, there may exist various possible reasons.
Revised: For word sense disambiguation tasks, there may exist various possible reasons.

Original: Since GPT-3 also performs like a random choice, Brown et al.
Revised: Since GPT-3 also performs like a random choice, Brown et al.

Original: (2020)) argues that this may be due to GPT-3’s disability of the bidirectionality.
Revised: (2020)) argues that this may be due to GPT-3’s disability of the bidirectionality.

Original: However, even though RoBERTa is bidirectional, it also fails to achieve promising results.
Revised: However, even though RoBERTa is bidirectional, it also fails to achieve promising results.

Original: Model scaling-up may be another reason, but no BERT-like model has comparable scaling-up to GPT-3 to verify this point.
Revised: Model scaling-up may be another reason, but no BERT-like model has comparable scaling-up to GPT-3 to verify this point.

Original: Besides, since prompt can help PLM recall the data knowledge during pre-training (Zhong et al., 2021), another possible reason is that there is no data similar to the task in pre-training.
Revised: Besides, since prompt can help PLM recall the data knowledge during pre-training (Zhong et al., 2021), another possible reason is that there is no data similar to the task in pre-training.

Original: 5. Z ERO - SHOT P ROMPT - BASED M ETHODS V . S . F EW - SHOT L EARNING M ETHODS
Revised: 5. Z ERO - SHOT P ROMPT - BASED M ETHODS V . S . F EW - SHOT L EARNING M ETHODS

Original: Comparing zero-shot prompt-based methods to few-shot learning methods on various tasks, we ﬁnd that few-shot learning can improve performance considerably.
Revised: Comparing zero-shot prompt-based methods to few-shot learning methods on various tasks, we ﬁnd that few-shot learning can improve performance considerably.

Original: Therefore, although zero-shot prompt-based methods achieve promising performance on some tasks, there is still a big gap between zero-shot prompt-based methods and few-shot learning methods, which shows that the current PLMs still need more supervision than label names to deal with various downstream tasks.
Revised: Therefore, although zero-shot prompt-based methods achieve promising performance on some tasks, there is still a big gap between zero-shot prompt-based methods and few-shot learning methods, which shows that the current PLMs still need more supervision than label names to deal with various downstream tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The few-shot and zero-shot capabilities of GTM series (Radford et al., a;b; Brown et al., 2020) make prompt-based learning draw signiﬁcant attention in the ﬁeld of large-size language model pretraining and adaptation.
Revised: The few-shot and zero-shot capabilities of GTM series (Radford et al., a;b; Brown et al., 2020) make prompt-based learning draw signiﬁcant attention in the ﬁeld of large-size language model pretraining and adaptation.

Original: Existing works of prompt-based learning are mainly from the following two angles, i.e., prompt engineering and answer engineering 6 .
Revised: Existing works of prompt-based learning are mainly from the following two angles, i.e., prompt engineering and answer engineering 6 .

Original: [Table]
Revised: [Table]

Original: Prompt Engineering Prompt engineering aims to construct a better prompt function to improve the performance of the downstream tasks.
Revised: Prompt Engineering Prompt engineering aims to construct a better prompt function to improve the performance of the downstream tasks.

Original: To this end, Petroni et al.
Revised: To this end, Petroni et al.

Original: ; Brown et al.
Revised: ; Brown et al.

Original: Schick & Sch ¨ utze (2021a;b) create prompt manually to handle various tasks, including knowledge probing, question answering, translation, and text classiﬁcation.
Revised: Schick & Sch ¨ utze (2021a;b) create prompt manually to handle various tasks, including knowledge probing, question answering, translation, and text classiﬁcation.

Original: However, the handcrafted prompts need lots of human effort and may fail to be optimal (Jiang et al., 2020).
Revised: However, the handcrafted prompts need lots of human effort and may fail to be optimal (Jiang et al., 2020).

Original: Many work focus on achieving prompts automatically, which can be divided into discrete (Davison et al., 2019; Jiang et al., 2020; Haviv et al., 2021; Shin et al., 2020;
Revised: Many work focus on achieving prompts automatically, which can be divided into discrete (Davison et al., 2019; Jiang et al., 2020; Haviv et al., 2021; Shin et al., 2020;

Original: Gao et al., 2021) and continuous (Zhong et al., 2021; Qin & Eisner, 2021; Li & Liang, 2021; Lester et al., 2021;
Revised: Gao et al., 2021) and continuous (Zhong et al., 2021; Qin & Eisner, 2021; Li & Liang, 2021; Lester et al., 2021;

Original: Hambardzumyan et al., 2021; Liu et al., 2021b; Zhang et al., 2021), to avoid these problems.
Revised: Hambardzumyan et al., 2021; Liu et al., 2021b; Zhang et al., 2021), to avoid these problems.

Original: In recent works, Logan IV et al.
Revised: In recent works, Logan IV et al.

Original: (2021) proposed null prompts , which contain no task-speciﬁc prompt, and achieve comparable performance than handcrafted prompts.
Revised: (2021) proposed null prompts , which contain no task-speciﬁc prompt, and achieve comparable performance than handcrafted prompts.

Original: One parallel work with us is Sanh et al.
Revised: One parallel work with us is Sanh et al.

Original: (2021), which utilizes multi-task training with multiple prompts to test the generalization ability on zero-shot task learning.
Revised: (2021), which utilizes multi-task training with multiple prompts to test the generalization ability on zero-shot task learning.

Original: Unlike previous works that require labeled data to select prompts or post-tune the model parameters, we launch an empirical study to learn the potential and limitations of prompt-based methods under the real zero-shot setting, i.e., without labeled data for prompt selection and tuning.
Revised: Unlike previous works that require labeled data to select prompts or post-tune the model parameters, we launch an empirical study to learn the potential and limitations of prompt-based methods under the real zero-shot setting, i.e., without labeled data for prompt selection and tuning.

Original: Answer Engineering Answer engineering selects label words and constructs a verbalizer to make the model perform better.
Revised: Answer Engineering Answer engineering selects label words and constructs a verbalizer to make the model perform better.

Original: Meng et al.
Revised: Meng et al.

Original: (2020) use unlabeled data to search candidate label words.
Revised: (2020) use unlabeled data to search candidate label words.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2020) paraphrase label words to broaden its space.
Revised: (2020) paraphrase label words to broaden its space.

Original: Schick et al.
Revised: Schick et al.

Original: (2020) use small amounts of training data to compute a word’s suitability as label words iteratively.
Revised: (2020) use small amounts of training data to compute a word’s suitability as label words iteratively.

Original: Shin et al.
Revised: Shin et al.

Original: (2020) learn a logistic classiﬁer to select label word.
Revised: (2020) learn a logistic classiﬁer to select label word.

Original: Gao et al.
Revised: Gao et al.

Original: ﬁrst use PLMs to select the candidate label words and then extract the ﬁnal label words based on their zero-shot performance.
Revised: ﬁrst use PLMs to select the candidate label words and then extract the ﬁnal label words based on their zero-shot performance.

Original: Hambardzumyan et al.
Revised: Hambardzumyan et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021) learn a continuous label words.
Revised: (2021) learn a continuous label words.

Original: Hu et al.
Revised: Hu et al.

Original: (2021) use external knowledge to expand the label word space of the verbalizer.
Revised: (2021) use external knowledge to expand the label word space of the verbalizer.

Original: In our work, we focus on making full of the ability of PLMs without using any extra knowledge and labeled/unlabeled data.
Revised: In our work, we focus on making full of the ability of PLMs without using any extra knowledge and labeled/unlabeled data.

Original: However, all these works need unlabeled data, which may not be available in many realistic scenarios.
Revised: However, all these works need unlabeled data, which may not be available in many realistic scenarios.

Original: In this work, we focus on the zero-shot potential of the prompting paradigm.
Revised: In this work, we focus on the zero-shot potential of the prompting paradigm.

Original: Different from previous work, we do not utilize unlabeled data and only introduce limited costs for label names.
Revised: Different from previous work, we do not utilize unlabeled data and only introduce limited costs for label names.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper thoroughly studies the effectiveness of powerful few-shot learning methods on pretrained language models from the BERT family in the zero-shot setting.
Revised: This paper thoroughly studies the effectiveness of powerful few-shot learning methods on pretrained language models from the BERT family in the zero-shot setting.

Original: We surprisingly ﬁnd that simply introducing a few prompt [MASK]s could signiﬁcantly improve the performance and robustness of the Null Prompt method and even exceed cherry-piked manual prompts, which shows the zero-shot potential of BERT family.
Revised: We surprisingly ﬁnd that simply introducing a few prompt [MASK]s could signiﬁcantly improve the performance and robustness of the Null Prompt method and even exceed cherry-piked manual prompts, which shows the zero-shot potential of BERT family.

Original: We also launched a coarse-to-ﬁne study to learn the inﬂuence of multiple components in our proposed method.
Revised: We also launched a coarse-to-ﬁne study to learn the inﬂuence of multiple components in our proposed method.

Original: In the end, we brieﬂy discuss the possible limitations of current zero-shot methods.
Revised: In the end, we brieﬂy discuss the possible limitations of current zero-shot methods.

Original: In the near future, we will explore more under-performed tasks.
Revised: In the near future, we will explore more under-performed tasks.

Original: A APPENDIX A. D ETAILED E XPERIMENT S ETTINGS
Revised: A APPENDIX A. D ETAILED E XPERIMENT S ETTINGS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ﬁlm
Revised: ﬁlm

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [ X ]
Revised: [ X ]

Original: [Table]
Revised: [Table]

Original: is [MASK]
Revised: is [MASK]

Original: [Table]
Revised: [Table]

Original: [ X ] This text shows [MASK] sentiment
Revised: [ X ] This text shows [MASK] sentiment

Original: [Table]
Revised: [Table]

Original: [ X ] The attitude of this text is [MASK]
Revised: [ X ] The attitude of this text is [MASK]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [MASK]
Revised: [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: This
Revised: This

Original: [Table]
Revised: [Table]

Original: shows [MASK]
Revised: shows [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ] The attitude of this text is [MASK]
Revised: ] The attitude of this text is [MASK]

Original: [Table]
Revised: [Table]

Original: The next topic is [MASK] .
Revised: The next topic is [MASK] .

Original: [ X ]
Revised: [ X ]

Original: [Table]
Revised: [Table]

Original: The following is about [MASK] .
Revised: The following is about [MASK] .

Original: [ X ]
Revised: [ X ]

Original: [Table]
Revised: [Table]

Original: [ X ] The topic of this text is [MASK] .
Revised: [ X ] The topic of this text is [MASK] .

Original: DBPedia
Revised: DBPedia

Original: The next topic is [MASK] .
Revised: The next topic is [MASK] .

Original: [ X ] 60.
Revised: [ X ] 60.

Original: The following is about [MASK] .
Revised: The following is about [MASK] .

Original: [ X ] 73.
Revised: [ X ] 73.

Original: [ X ] The topic of this text is [MASK] .
Revised: [ X ] The topic of this text is [MASK] .

Original: Table 7 reports the label names for IMDB, Amazon, AG News, DBPedia.
Revised: Table 7 reports the label names for IMDB, Amazon, AG News, DBPedia.

Original: Table 8 and Table 9 show the templates for IMDB, Amazon, AG News, DBPedia, which are modiﬁed from Schick & Sch¨utze (2021b), and theirs accuracy of Manual Prompt (Prior) and Manual Prompt (w/o Engineering), respectively.
Revised: Table 8 and Table 9 show the templates for IMDB, Amazon, AG News, DBPedia, which are modiﬁed from Schick & Sch¨utze (2021b), and theirs accuracy of Manual Prompt (Prior) and Manual Prompt (w/o Engineering), respectively.

Original: Table 10 shows the templates we used for NSP-BERT, which is modiﬁed from Yin et al.
Revised: Table 10 shows the templates we used for NSP-BERT, which is modiﬁed from Yin et al.

Original: (2019), and their accuracy.
Revised: (2019), and their accuracy.

Original: Table 11 shows the shows the templates we used for GLUE and SuperGLUE, which is introduced from Gao et al.
Revised: Table 11 shows the shows the templates we used for GLUE and SuperGLUE, which is introduced from Gao et al.

Original: (2021) and Schick & Sch¨utze (2021b).
Revised: (2021) and Schick & Sch¨utze (2021b).

Original: For SST-2,
Revised: For SST-2,

Original: MNLI, QNLI, RTE, we report accuracy; for MRPC and QQP, we report F1; for CoLA, we report
Revised: MNLI, QNLI, RTE, we report accuracy; for MRPC and QQP, we report F1; for CoLA, we report

Original: Matthew’s correlation; for STS-B, we report Pearson’s correlation.
Revised: Matthew’s correlation; for STS-B, we report Pearson’s correlation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [ X ]
Revised: [ X ]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [MASK]
Revised: [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ] [MASK]
Revised: ] [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [ X 1 ] [MASK], [
Revised: [ X 1 ] [MASK], [

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [ P ]
Revised: [ P ]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [MASK]
Revised: [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [ P ].
Revised: [ P ].

Original: Question: [ Q ]?
Revised: Question: [ Q ]?

Original: Is it [ A ]?
Revised: Is it [ A ]?

Original: [MASK]
Revised: [MASK]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 ] [ X 2 ] Does [ W ] have the same meaning in both sentences?
Revised: 1 ] [ X 2 ] Does [ W ] have the same meaning in both sentences?

Original: [MASK]
Revised: [MASK]

Original: [Table]
Revised: [Table]

Original: S TRATEGY
Revised: S TRATEGY

Original: [Table]
Revised: [Table]

Original: Our proposed simple Multi-Null Prompt strategy can be effective for zero-shot text classiﬁcations, and the core spirit is to use multiple [MASK] tokens.
Revised: Our proposed simple Multi-Null Prompt strategy can be effective for zero-shot text classiﬁcations, and the core spirit is to use multiple [MASK] tokens.

Original: Therefore, to fully explore the potential of the strategy, we extend the veriﬁcation to more prompting strategies.
Revised: Therefore, to fully explore the potential of the strategy, we extend the veriﬁcation to more prompting strategies.

Original: That is, we combine the previous effective prompt methods with Multi-Null Prompt on the zero-shot setting to investigate its impact.
Revised: That is, we combine the previous effective prompt methods with Multi-Null Prompt on the zero-shot setting to investigate its impact.

Original: Speciﬁcally, we conduct experiments on our Multi-Null Prompt with (1) Manual Prompt (Prior), and (2) KPT (Hu et al., 2021) method on text classiﬁcations.
Revised: Speciﬁcally, we conduct experiments on our Multi-Null Prompt with (1) Manual Prompt (Prior), and (2) KPT (Hu et al., 2021) method on text classiﬁcations.

Original: The corresponding results are shown
Revised: The corresponding results are shown

Original: [Table]
Revised: [Table]

Original: in Table 12 and Table 13.
Revised: in Table 12 and Table 13.

Original: We can clearly observe that the performance for each task is further signiﬁcantly improved.
Revised: We can clearly observe that the performance for each task is further signiﬁcantly improved.

Original: For example, in Table 12, the accuracy on Amazon can be 93 .
Revised: For example, in Table 12, the accuracy on Amazon can be 93 .

Original: 50 , which is much better than 82 .
Revised: [Table] which is much better than 82 .

Original: 29 of Manual Prompt (Prior) and 86 .
Revised: 29 of Manual Prompt (Prior) and 86 .

Original: 22 of Multi-Null Prompt in Table 1.
Revised: 22 of Multi-Null Prompt in Table 1.

Original: The results clearly demonstrate the effectiveness of the combination of our multiple mask strategy and the other prompt strategies
Revised: The results clearly demonstrate the effectiveness of the combination of our multiple mask strategy and the other prompt strategies

Original: [Table]
Revised: [Table]

Original: A. E XAMPLES OF D IFFERENT P ROMPT S TRATEGIES
Revised: A. E XAMPLES OF D IFFERENT P ROMPT S TRATEGIES

Original: To better clearly compare the different prompt strategies used in this work
Revised: To better clearly compare the different prompt strategies used in this work

Original: [Table]
Revised: [Table]

Original: we put several examples of these strategies in Table
Revised: we put several examples of these strategies in Table

Original: [Table]
Revised: [Table]

Original: which are [CLS], Manual Prompt, Null Prompt, our Multi-Null
Revised: which are [CLS] [Table] Manual Prompt, Null Prompt, our Multi-Null

Original: Prompt, and NSP-BERT methods.
Revised: Prompt, and NSP-BERT methods.

Original: The differences can be found in these examples.
Revised: The differences can be found in these examples.

Original: From these examples, we can also see that our simple Multi-Null Prompt is complementary with others and can be easily combined with other strategies, which are also veriﬁed in the above study
Revised: From these examples, we can also see that our simple Multi-Null Prompt is complementary with others and can be easily combined with other strategies [Table] which are also veriﬁed in the above study

Original: [Table] [Table]
Revised: [Table] [Table]