Pre-trained Token-replaced Detection Model as Few-shot Learner (2022)

Abstract: Pre-trained masked language models have demonstrated remarkable ability as few-shot learners. In this paper, as an alternative, we propose a novel approach to few-shot learning with pre-trained token-replaced detection models like ELECTRA. In this approach, we reformulate a classification or a regression task as a token-replaced detection problem. Specifically, we first define a template and label description words for each task and put them into the input to form a natural language prompt. Then, we employ the pre-trained token-replaced detection model to predict which label description word is the most original (i.e., least replaced) among all label description words in the prompt. A systematic evaluation on 16 datasets demonstrates that our approach outperforms few-shot learners with pre-trained masked language models in both one-sentence and two-sentence learning tasks.

## introduction few-shot learning aims to learn models with a few examples and the learned models generalize well from very limited examples like humans. recently, few-shot learning has become an important and interesting research field of intelligence @xcite . compared to datarich supervised learning, few-shot learning greatly overcomes the expensive data annotation challenge in reality. some large pre-trained language models such as gpt-3 @xcite have achieved remarkable few-shot performance by reformulating tasks as language model problems. however, its hundreds of billions of parameters deter researchers and practitioners from applying it widely. to tackle this, a new paradigm, equipping smaller masked language models @xcite with fewshot capabilities @xcite has been explored, wherein downstream tasks are treated as cloze questions. typically, as illustrated in figure 1 (b), each input sentence is appended with a prompt phrase such as "it was @xcite " to each input sentence, allowing the model to fill in the [mask] by reusing the masked language model head. instead of masked language models, another selfsupervised pre-training task called token-replaced detection has been proposed by @xcite and it trains a model named electra to distinguish whether each token is replaced by a generated sample or not. one major advantage of token-replaced detection pre-training modeling is that it is more computationally efficient than masked language modeling. moreover, their research demonstrates that given the same model size, pre-trained token-replaced detection models achieve substantially better performance than the pre-trained masked language model such as bert @xcite and roberta @xcite in many downstream tasks. in this paper, inspired by the above unique effectiveness of the pre-trained token-replaced detection model, we propose a new approach, pre-trained token-replaced detection models as few-shot learners, aiming to further improve the few-shot learning performances. the key idea of our approach is to reformulate downstream tasks as token-replaced detection problems. specifically, we first define a template and label description words which will be used to convert the input sentence into a prompted text. then, we directly insert the template and all label description words into the sentence to form a prompt that might be an ungrammatical sentence. the motivation of this operation is to make our inputs similar to those in the data for training elec-tra, having some replaced tokens. lastly, we use a pre-trained token-replaced detection model to distinguish which label description word is the most original (i.e., least replaced) among all label description words. for instance, as illustrated in 1(c), when performing a sentiment classification task, the input sentence "i am so excited about the concert." is converted into a new one "i am so excited about the concert. it was great terrible" where "great" and "terrible" are two label description words for the two sentimental categories: positive and negative. consequently, the pre-trained token-replaced detection model may predict the label description word "great" is more original (i.e., less replaced) than "terrible", which indicates that the input belongs to a positive category. compared to few-shot learners with pre-trained masked language models, in general, there are two major differences as follows. first, the designed phrases of few-shot learners with pre-trained masked don't contain any label description word. however, in our approach, we put them in prompts directly, which is easier to understand. second, few-shot learners with pre-trained masked language models predict which label description word is the most appropriate to fill in @xcite , but our approach predicts which label description word is the most original (i.e., least replaced). to evaluate the few-shot capacity of our approach, we use both electra-base and electra-large as pre-trained token-replaced detection models to perform few-shot learning in our approach and conduct experiments in a wide variety of both one-sentence and two-sentence tasks. empirical studies demonstrate that our approach outperforms few-shot learners with pre-trained masked language models. the contributions of this study are as follows: the remainder of this paper is organized as follows. section 2 overviews related studies about few-shot learning approaches and pre-trained token-replaced detection models. section 3 proposes our few-shot learner with a pre-trained tokenreplaced detection model in detail. section 4 presents the experimental results and analysis. finally, section 5 discusses the conclusions and future work. ## our approach a pre-trained token-replaced detection model like electra @xcite trains a discriminator d that detects whether a token x t in an input token sequence @xmath0, . . . , x t , . . . , x n ] is an original or replaced one. suppose that the output from the discriminator is @xmath1, . . . , y t , . . . , y n ]. then, y @xmath2. specifically, in electra @xcite , the discriminator is trained together with a masked language modeling generator which is used to generate replaced tokens in a token sequence. finally, the discriminator performs the prediction with a sigmoid output layer, i.e., ## experiments in this section, we compare our approach with a few-shot learning approach based on pre-trained masked language models. furthermore, we evaluate the impact of different templates, label description words and training data scales. ## conclusion and future work in this paper, we propose a novel few-shot learning approach with pre-trained token-replaced detection models, which transforms traditional classification and regression tasks into token-replaced detection problems. empirical studies on 16 nlp datasets demonstrate that, in both one-sentence and two-sentence learning tasks, our approach generally achieves better performances in the few-shot scenario when compared to the masked language model-based few-shot learner. these results highlight that our approach is a comprehensive alternative for few-shot learning. in the future, we would like to explore the following directions. first, we notice that in some tasks like cola, standard fine-tuning is also a strong baseline and even performs much better than fewshot learners based on either a masked language model or a token-replaced detection model. thus, it is interesting to combine [cls] output vector, i.e., the standard fine-tuning style, with the prompt-ing style, to further improve the few-shot learning performance. second, we would like to apply our approach to some other nlp tasks, such as multilabel text classification and sequence labeling tasks like named entity recognition.