NSP - BERT : A Prompt-based Few-Shot Learner through an Original Pre-training Task —— Next Sentence Prediction (2022)

Abstract: Using prompts to utilize language models to perform various downstream tasks, also known as prompt-based learning or prompt-learning, has lately gained significant success in comparison to the pre-train and fine-tune paradigm. Nonetheless, virtually most prompt-based methods are token-level such as PET based on mask language model (MLM). In this paper, we attempt to accomplish several NLP tasks in the zero-shot and few-shot scenarios using a BERT original pre-training task abandoned by RoBERTa and other models——Next Sentence Prediction (NSP). Unlike token-level techniques, our sentence-level prompt-based method NSP-BERT does not need to fix the length of the prompt or the position to be predicted, allowing it to handle tasks such as entity linking with ease. NSP-BERT can be applied to a variety of tasks based on its properties. We present an NSP-tuning approach with binary cross-entropy loss for single-sentence classification tasks that is competitive compared to PET and EFL. By continuing to train BERT on RoBERTa’s corpus, the model’s performance improved significantly, which indicates that the pre-training corpus is another important determinant of few-shot besides model size and prompt method.

## introduction gpt-2 (up to 1.5b @xcite ) and @xcite ) are ultra-largescale language models with billions of parameters that have recently demonstrated outstanding performance in various nlp tasks. compared with previous state-of-the-art fine-tuning methods, they can achieve competitive results without any or with just a limited quantity of training data. although classification @xcite [eos] every [blank] has two side. ## conclusion in this paper, we show that nsp can also be an apposite zero-shot or few-shot learner same as mlm. this not only provides a new route for promptlearning, but also makes us rethink the role of sentence-level pre-training tasks. at the same time, we continue to pre-train the bert model with a small amount of computing power, and its performance improves significantly on both zero-shot and few-shot learning, whether to use pet or nsp-bert. we believe that not only the size of the model, but also the pre-training corpus, both determine the upper limit of the model's ability on few-shot learning.