Automated Few-Shot Classification with Instruction-Finetuned Language Models (2023)

Abstract: A particularly successful class of approaches for few-shot learning combines language models with prompts - hand-crafted task descriptions that complement data samples. However, designing prompts by hand for each task commonly requires domain knowledge and substantial guesswork. We observe, in the context of classification tasks, that instruction finetuned language models are remarkably robust towards some dimensions of a prompt’s design. We subsequently propose a simple method to eliminate the need for handcrafted prompts, named AuT-Few. This approach consists of (i) a prompt retrieval module that selects suitable task instructions from the instruction-tuning knowledge base, and (ii) the generation of two distinct, semantically meaningful, class descriptions and a selection mechanism via cross-validation. Over 12 datasets, spanning 8 classification tasks, we show that AuT-Few outperforms current state-of-the-art few-shot learning methods. Moreover, AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark. Notably, these results are achieved without task-specific handcrafted prompts on unseen tasks.

## introduction collecting annotated data is time-consuming and expensive. the goal of few-shot learning is to address this limitation by developing models that generalize from a small number of training examples. a now dominant paradigm in few-shot learning involves pre-training a large language model (plm) on unsupervised language modelling objectives, combined with supervised fine-tuning @xcite . fine-tuning on a variety of classification tasks improves generalization to new unseen tasks even further @xcite @xcite . prompts, instructions that describe the tasks in natural language, are crucial to successful finetuning on many tasks. typically, prompts consist of two components: task templates and answer choices. task templates are textual instructions about the task. answer choices are semantic descriptions of the categorical labels. supervised training on prompted samples, as shown in figure 1 , helps plms generalize when instructed via prompts on a new problem (here natural language inference). following lin et al. (2022), we use the term upstream model for these instructionfinetuned plms. these prompted upstream models provide state-of-the-art few-shot learning ( @xcite ), yet they still rely on strenuous manual intervention from manually crafted prompts, designed by experts with domain knowledge about the underlying tasks. task template answer choices instruction tuning prompts sentiment review: we came here on saturday night [...] how does the reviewer feel about the movie? 0: very negative [...] 5: very positive paraph. last year, comcast [...] is that a paraphrase of the sentence comcast has about [...]. 0: yes 1: no unseen target task prompts nli given oil prices fall back as yukos oil threat lifted. can we guarantee that oil prices rise. is true? 0: yes 1: no in this paper, we are concerned with an automated few-shot classification regime, where the algorithm can only access the training samples oil prices fall back as yukos oil threat lifted. is that a paraphrase of the following sentence: topic-specific (dataset default) template-tailored paraphrase iden. and their categorical labels. while efforts have been made to automate prompting, these methods are not directly transferable to upstream models. most techniques target prompted masked language models (i.e. encoder-only models, that make predictions over continuous embeddings via its mask token @xcite . automation methods for models with a discrete output space (i.e. a decoder over the vocabulary) are costly and limited to the automation of the task template, still relying on handcrafted descriptions of labels @xcite . to automate few-shot learning with upstream models, we analyse the role of prompts across various classification tasks and we observe that upstream models exhibit low variability towards task-unspecific templates. in contrast, the selection of suitable answer choices can be important, yet answer choices do not need to be tailored to the specific instruction (e.g. yes/no for a polar question). these insights confirm observations by @xcite in a broader context and they motivate a simple few-shot learning automation method for upstream models, named aut-few. aut-few builds on the state-of-the-art learning method t-few @xcite , but crucially does not use any task-specific handcrafted prompts. aut-few automatically finds the most relevant templates to our target task from the collection prompts used to instruction-tune the upstream model. as illustrated in figure 2 , given an nli task, aut-few might retrieve templates written for paraphrase identification. to automate answer choices, aut-few generates label descriptions tailored to the retrieved templates (e.g., yes/no for a polar question, as for the illustrated paraphrase identification template) and descriptions that capture a class' overall topic (e.g. enron/purchase for enron spam classification). aut-few selects the most appropriate configuration via cross-validation. aut-few outperforms strong baselines, including t-few @xcite , by 2.1 points over a total of 12 datasets, spanning 8 tasks, without any task-specific handcrafted prompts. all but one task are unseen to the upstream models, indicating aut-few's strong generalization capabilities. moreover, by applying aut-few to a small upstream model (bart0 @xcite )), we achieve competitive performance and efficiency to the current state-ofthe-art prompt-free method, setfit @xcite . furthermore, aut-few achieves the best average rank across datasets on the few-shot raft benchmark @xcite . an ablation justifies the components of our automation method. foot_0 2 background and related work ## how much does the design of prompts matter for upstream models? to automate prompts, we need to understand their role in few-shot classification. while previous research suggests that the wording of instructions for masked language models is crucial, @xcite observe that the semantic relevance of a prompt is not a strong performance indicator for upstream models. however, their analysis is restricted to natural language inference whilst using the pet @xcite algorithm to train the model. yet, results in schick and schütze (2022) suggest that templates do matter in principle, but pet is robust when correctly configured. these results raise questions regarding the role of prompts for upstream models in the context of automated few-shot learning on unseen tasks. we conduct a systematic ablation study for both templates φ and answer choices ψ. we use t-few with the t0 upstream model and 32 samples per class. we evaluate 12 datasets, spanning 8 tasks. for details on the datasets, see appendix a. templates. we design four experiments to understand the importance of accurate task descriptions (i.e. semantics) in increasing order: concatenation of a sample's content without any additional text (null), uniform sampling of words from the training vocabulary (random), general purpose instructions (e.g. given ..., the answer is ...) that are not tailored to the task (general), handcrafted instructions (handcrafted). we use the same handcrafted answer choices and templates across all settings (and vice versa the same templates across experiments for answer choice experiments). as seen in figure 3 (top), with a mean score of 62.8, 62.9, 64.0, 64.2, for each setting, respectively, we observe that simple task-unspecific templates perform surprisingly well, only performing slightly worse than more complex handcrafted ones. templates that are not well-formed or lack an instruction entirely perform substantially worse than handcrafted ones. note that results differ heavily between datasets. while some datasets (enron and cr) are virtually unaffected by the design of the template, performance is strongly affected by the template for some other (e.g. rte, wsc, amazon). ## aut-few: automated few-shot classification with upstream models aut-few is a simple, yet efficient, algorithm to automate prompts for upstream models, drawing from the insights gained from section 3. figure 2 shows an illustration of aut-few's template and answer choice automation. aut-few deploys a lightweight template automation approach since accurate task templates are not essential to performance. it selects suitable templates from the collection of prompts the upstream model was instructionfinetuned on (section 4.1). on the other hand, the selection of answer choices has a substantial impact on performance. searching over all possible answer choices is intractable for large upstream models and also imprecise due to the small training size. thus, aut-few only considers two distinct types of answer choices (section 4.2). one is tailored to the retrieved templates by measuring the log-likelihood on the training data (template-tailored). the other is based on capturing the topic of samples belonging to the same class (topic-specific). we select the most appropriate template and answer choice configurations via cross-validation. the automated prompts are then used for training and inference of our upstream model, where we largely follow t-few (c.f. section 5.1 for details). ## conclusion aut-few replaces hand-designed task-specific prompts with automated templates, and achieves state-of-the-art results on a wide range of datasets