Prompting ELECTRA : Few-Shot Learning with Discriminative Pre-Trained Models (2022)

Abstract: Pre-trained masked language models successfully perform few-shot learning by formulating downstream tasks as text infilling. How- ever, as a strong alternative in full-shot settings, discriminative pre-trained models like ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks. ELECTRA is pre-trained to distinguish if a token is generated or original. We naturally extend that to prompt-based few-shot learning by training to score the originality of the target options without introducing new parameters. Our method can be easily adapted to tasks involving multi-token predictions without extra computation overhead. Analysis shows that ELECTRA learns distributions that align better with downstream tasks.

## introduction large pre-trained language models are known to be effective zero-shot and few-shot learners when scaled @xcite @xcite . much smaller masked language models (mlms), like bert @xcite and roberta @xcite , can be fined-tuned with only a few examples by utilizing prompt-based fine-tuning, which updates the model to select the correct target word or option @xcite . in this paper, we hypothesize that discriminative pre-trained models like electra @xcite will make even stronger few-shot learners as alternatives to mlms as they are pre-trained to distinguish between challenging alternatives. to test this hypothesis, we explore prompt-based learning with electra by aligning its pre-training 1 code is available at https://github.com/faceb ookresearch/electra-fewshot-learning . c( ) [mask] it is pretty damned funny . it is pretty damned funny . i ate an apple pie so @xcite and copa @xcite . the underlined text is the task-specific template. c(•): contextualized embedding; y and y ′ : a correct and an incorrect option, respectively. we propose an approach to prompting elec-tra, as shown in figure 1 . though trained with the same or even less computation than bert and roberta, electra turns out to be a more effective few-shot learner. it outperforms bert and roberta by 10.2 and 3.1 points on average across 9 tasks with single-token options for base-sized models in the few-shot setting, and the trend prevails for large-sized models. electra also outperforms roberta on 4 tasks with multi-token options. our analysis suggests that the failing predictions from electra's generator could actually feed negatives with opposite meanings from the correct tokens to the discriminator, which strengthens electra's ability to distinguish concepts with opposite meanings for zero-shot predictions. ## conclusion we explore discriminative pre-trained models for prompt-based zero-shot and few-shot learning. we find that these models consistently outperform masked language models that are trained with equivalent or even less computation, suggesting that discriminative pre-trained models are more effective zero-shot and few-shot learners. analysis shows that the electra's generator could very likely feeds negatives like antonyms to the discriminator, which serves as a direct contrast during pre-training. we also speculate that discriminative models are less vulnerable to the surface form competition @xcite , and we would like to dig deeper into this hypothesis in future work.