Prompt-Based Meta-Learning For Few-shot Text Classification (2022)

Abstract: Few-shot Text Classification predicts the semantic label of a given text with a handful of supporting instances. Current meta-learning methods have achieved satisfying results in various few-shot situations. Still, they often require a large amount of data to construct many few-shot tasks for meta-training, which is not practical in real-world few-shot scenarios. Prompt-tuning has recently proved to be another effective few-shot learner by bridging the gap between pre-train and downstream tasks. In this work, we closely combine the two promising few-shot learning methodologies in structure and propose a Prompt-Based Meta-Learning (PBML) model to overcome the above meta-learning problem by adding the prompting mechanism. PBML assigns label word learning to base-learners and template learning to meta-learner, respectively. Experimental results show state-of-the-art performance on four text classification datasets under few-shot settings, with higher accuracy and good robustness. We demonstrate through low-resource experiments that our method alleviates the shortcoming that meta-learning requires too much data for meta-training. In the end, we use the visualization to interpret and verify that the meta-learning framework can help the prompting method converge better. We release our code to reproduce our experiments.

## introduction humans can quickly learn new knowledge from a few examples, reflecting a high degree of intelligence. the meta-learning method was proposed to implement such human-like intelligence and achieved promising results in various few-shot situations. it lifts the training unit from data point to task and trains a meta-learner over many tasks (called episodes) to grasp the few-shot learning the original text with a template. then a pre-trained language model (plm) will fill in @xcite with words from a predefined label word set. a verbalizer will further map the predicted label word into the corresponding task label. given that the above two approaches both achieved satisfying results in various few-shot situations, we hope to alleviate the shortcomings of meta-learning methods by adding a prompting mechanism. in this work, we use an elegant way to closely combine these two approaches in structure and propose prompt-based meta-learning (pbml) to push the performance on few-shot text classification. in terms of prompts, we adopt the "soft" strategy (i.e., use continuously differentiable label words and templates). regarding meta-learning, our meta-learner mainly learns soft template embeddings and an mlm-based encoder. the core idea of our combination of the two approaches is to assign template&encoder learning to the meta-learner and label word learning to baselearners, respectively. the intuition is that different tasks may involve different classes, and label words need to consider specific classes, so the learning of label words is handled over to base-learners for task-specific adaptation. correspondingly, the output embedding at [mask] position by the prompting method reflects the model's understanding of the text. various tasks should share this ability of natural language understanding (nlu), which is why the learning of template and encoder is assigned to the meta-learner. experimental results demonstrate the good compatibility between meta-learning and prompttuning. our pbml (1) achieved a new state-of-theart performance on four datasets under few-shot settings, with higher accuracy and good robustness. (2) through low-resource experiments, we can clearly see that the prompting mechanism helps pbml maintain good performance when the available meta-training data is significantly reduced. (3) comparison with prompt-tuning baseline demonstrates that meta-learning enables prompt-tuning to adapt more efficiently to new tasks. (4) the visualization results verify that the "learning-to-learn" framework could result in better convergence for prompts. ## methodology this section presents the details of our proposed framework, pbml (figure 2 ). pbml consists of 3 parts: firstly, the meta-learner (meta-encoder) encodes instances and gets the embedding of the [mask] token for each instance. secondly, we explore an external knowledge graph for continuous label word initialization. then a base-learner will update the label word embeddings using predicted embeddings of support instances. inference for queries is based on the adapted label word embeddings, and we use the loss on the query set q for meta-optimization. ## experiments this section evaluates the performance of our proposed pbml. we conduct extensive experiments on four widely-used text classification datasets under few-shot settings and make a full-scale comparison with existing state-of-the-art baselines. we report our implementation details in appendix e. ## conclusion in conclusion, we propose prompt-based metalearning for few-shot text classification. our method combines prompt-tuning with metalearning closely, where the base learners tune task-specific label words (learn) and the metalearner tunes the task-agnostic template and encoder (learning-to-learn). experimental results demonstrate the excellent compatibility of metalearning and prompt-tuning, with state-of-the-art performance in various few-shot text classification tasks and good robustness under noisy settings. additionally, we show that the prompting mech-anism helps pbml maintain good performance with significantly reduced meta-training data. the visualization results further demonstrate that metalearning enables prompt-tuning to distinguish semantics better.