Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022)

Abstract: Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.

## introduction recent work has obtained strong zero-shot results by prompting language models @xcite . as formalized by @xcite , the core idea is to reformulate text classification as language modeling using a pattern and a verbalizer. given the input space x, the output space c and the space of possible strings v * , the pattern t : @xmath0, whereas the verbalizer v : @xmath1. a language model can then be used for zero-shot classification by picking the most likely completion for the masked text arg max @xmath2. 2 in it was not funny at all. the actor played well. ## proposed method as shown in figure 1 , our method has three steps: mine. we first use the pattern and a set of verbalizers to extract labeled examples from the corpus. to that end, we define patterns that are filled with verbalizers and expanded into regular expressions. for instance, the pattern and verbalizer in figure 1 would extract every sentence following "is good." or "was good." as an example of the positive class, and every sentence following "is bad." or "was bad." as an example of the negative class. in practice, the patterns that we define are comparable to the ones used for prompting, and the verbalizers are exactly the same (see tables 1 and 2 ). appendix a gives more details on how we expand patterns into regular expressions. while prior work in prompting typically uses a single verbalizer per class, our approach allows to naturally combine examples mined through multiple verbalizers in a single dataset. so as to mitigate class imbalance and keep the mined dataset to a reasonable size, we mine a maximum of 40k examples per class after balancing across the different verbalizers. filter. as an optional second step, we explore automatically removing noisy examples from the mined data. to that end, we classify the mined examples using zero-shot prompting, and remove examples for which the predicted and the mined label do not match. this filtering step is reliant on the performance of prompting, and we only remove 10% of the mismatching examples for which zeroshot prompting is the most confident. finetune. finally, we use the mined dataset to finetune a pretrained language model in the standard supervised fashion @xcite , learning a new classification head. ## experimental settings tasks. we evaluate on three types of tasks: binary sentiment analysis on amazon @xcite 1 ). we report average accuracy across 3 runs for all systems except prompting. w/ multi verb.: prompting with different sets of verbalizers ( approaches. we compare the following methods in our experiments, using roberta-base @xcite as the pretrained model in all cases: for the fine-tuning step, we use the same settings as in the full-shot setup, except that we train for 5,000 steps with a dropout probability of 0.4. 5 to mitigate class imbalance, we form batches by first sampling the class for each instance from the uniform distribution, and then picking a random example from the mined data belonging to that class. patterns and verbalizers. we use comparable patterns for prompting and mining with the exact same verbalizers, which we report in pattern verbalizer avg prompting {verbalizer} stars: {input} 5 / 1 51.0 {input} i {verbalizer} it. love / hate 73.1 {input} it is {verbalizer}. good / bad 78.1 mining {verbalizer} star*. {input} 5 / 1 72.1 i {verbalizer}*. {input} love / hate 85.5 (is|was) {verbalizer}*. {input} good / bad 72.0 ## results and analysis we next discuss our main findings and report additional results in appendix c. ## related work recent work in zero-shot learning has explored a similar generate-filter-finetune approach, but using large language models instead of mining to generate training data @xcite @xcite . mining-based approaches have a long tradition in information extraction @xcite . however, to the best of our knowledge, we are the first to apply them for zero-shot learning as an alternative to prompting. instead of mining examples for the target task, bansal et al. ## conclusions in this work, we have shown that mining-based zero-shot learning outperforms prompting. moreover, our approach shows headroom for further improvement by exploring filtering techniques. the flexibility of our approach enables additional directions like domain filtering, bootstrapping, and interactive pattern/verbalizer design, where practitioners would inspect a few mined examples and refine their patterns until they are satisfied. in addition, our methods can serve as a partial explanation for why prompting works, showing that task-relevant examples are often present in the pretraining corpus in an explicit form, to the extent that they can be directly mined through simple regular expressions. nevertheless, we believe that there can be other factors involved, as evidenced by the best patterns and verbalizers being different for mining and prompting, and we believe that delving deeper into the relation between pretraining data and prompting performance is an interesting future direction.