Learning New Tasks from a Few Examples with Soft-Label Prototypes (2024)

Abstract: Existing approaches to few-shot learning in NLP rely on large language models (LLMs) and/or fine-tuning of these to generalise on out-of-distribution data. In this work, we propose a novel few-shot learning approach based on soft-label prototypes (SLPs) designed to collectively capture the distribution of different classes across the input domain space. We focus on learning previously unseen NLP tasks from very few examples (4, 8, 16) per class and experimentally demonstrate that our approach achieves superior performance on the majority of tested tasks in this data-lean setting while being highly parameter efficient. We also show that our few-shot adaptation method can be integrated into more generalised learning settings, primarily meta-learning, to yield superior performance against strong baselines.

## introduction humans have a remarkable ability to adapt knowledge gained in one domain and apply it in another setting, and to identify or disambiguate objects after observing only a handful of examples @xcite . this has inspired research in few-shot learning that aims to build models that can learn a new task using only a small number of examples per class. early few-shot learning in nlp relied on interventions at the data level, such as dataset augmentation @xcite or generation of adversarial examples from few-shot datasets @xcite , while more recent approaches @xcite utilise meta-learning @xcite to optimise model parameters such that models adapt quickly to new tasks using past experience @xcite @xcite . the advent of large language models (llms) has led to a plethora of further methods, including fine-tuning on different target tasks @xcite , cre-ating prompt-enhanced few-shot datasets for training @xcite @xcite as well as parameter-efficient fine-tuning methods for very large language models, with parameters running into billions @xcite . in this paper, we propose a simple and effective approach to few-shot learning based on soft-label prototypes (slps) that capture the distribution of different classes across the input domain space, inspired by previous work on generating compact representations of input training data @xcite . our contributions are summarised as follows: 1) we develop a novel neural framework for few-shot learning via soft-label prototypes that has a very small computational and memory footprint, and achieves state-of-the-art results in limited-resource settings. our approach (deepslp) does not rely on (expensive) llm parameter updates or auxiliary training data. 2) we focus on few-shot learning of new, unseen nlp tasks using as little as 4 examples per class, and demonstrate that we outperform strong baselines on the majority of test tasks. 3) we demonstrate that our approach can also be effectively adapted (metaslp) in highresource settings when auxiliary training data is available for few-shot learning, and performs competitively when compared against strong baselines. 4) we release our code and data to facilitate further research in the field. 1 ## related work early few-shot learning approaches in nlp include data augmentation and semi-supervised learning; e.g., augmentation with adversarial examples @xcite , interpolation of training data into a learnable higher dimensional embedding space @xcite , and consistency training to make models more resistant to noise @xcite . recent research efforts on large-scale pretraining of language models @xcite @xcite @xcite reduce the amount of data required for their subsequent fine-tuning or utilisation in a given task. instruction tuning and in-context learning @xcite @xcite @xcite @xcite show that natural language instructions or prompts can enhance a model's few-shot learning abilities by leveraging the language (instruction) understanding abilities of the given pretrained llm @xcite . to finetune extremely large language models (with billions of parameters) efficiently, a host of parameterefficient fine-tuning techniques have also been @xcite , which leverage pre-trained llms and produce superior results on tasks such as question answering, reasoning, text summarisation, coding, etc. @xcite @xcite . however, the search space over llms, prompt templates and few-shot learning is so great that there is yet to be an established standard. different models require different styles of (few-shot) prompting, and certain prompt templates may work better with specific llms and datasets rather than universally across the board (e.g., @xcite ). furthermore, evaluating robustness of state-of-the-art / generative llms on new, unseen tasks (ood generalisation) presents a significant challenge due to their vast and unknown training data, resulting in artificially inflated performance as a result of data leakage @xcite . previous work has also tackled few-shot learning within the meta-learning paradigm of "learning to learn" @xcite @xcite , utilising methods that are trained to adapt quickly (in a few gradient steps) to new tasks and from a small number of examples, using past experience. meta-learning has emerged as a promising technique for a range of tasks @xcite @xcite , including nlp such as natural language inference, text classification, etc. @xcite @xcite @xcite @xcite . in a similar spirit to parameter-efficient fine-tuning @xcite , our work shifts away from the aforementioned paradigms that suffer from lack of standardisation (llm few-shot prompting) and increased computational complexity for fine-tuning (e.g., fine-tuning extremely large language models such as gpt @xcite . we present a novel, parameterefficient few-shot learning framework (deepslp) based on soft-label prototypes (slps), which we show to be effective on a range of tasks in limited and high-resource settings, while having a substantially smaller computational and memory footprint. while deepslp does not rely on llm fine-tuning or auxiliary training data, we present a variant (metaslp) that can be used for few-shot learning via auxiliary data and fine-tuned encoders. we target few-shot learning of new, unseen tasks (i.e., tasks and classes not previously trained on) (a) in limited-resource settings, without access to auxiliary training data, and with frozen model parameters, and (b) in high-resource settings, with access to auxiliary training data, which are used to update model parameters. for the latter, our work is similar to @xcite . the authors target few-shot learning of unseen tasks via meta-learning, utilising auxiliary training data. ## approach: few-shot learning with soft-label prototypes (slps) a soft label is defined as a vector y representing a data point's simultaneous membership to several classes @xcite , essentially denoting a point's partial association to different classes. using this definition, a soft-label prototype (slp) is defined as ( ⃗ x, y ), where ⃗ x is a point in input space (e.g., an input feature vector) and y is its corresponding soft label. the underlying idea in sucholutsky and schonlau (2021)'s work is that a small set of soft-label prototypes can be used to accurately represent a training set. we build on this idea and reframe slps for the task of few-shot learning of new, unseen tasks where very small amounts of data are available per class. ## experimental settings and datasets experimental settings we experiment with two settings in terms of amounts of available data. the first is a limited-resource setting where we only train / fine-tune our models in a few-shot manner on small amounts of training data (i.e., in the absence of auxiliary training data). the other setting is a high-resource setting where we assume that auxiliary training data is available for additional training / fine-tuning. ## baselines our aim is to determine how well our models -deepslp and metaslp -perform in the low and high-resource setting respectively, compared to strong baselines when given the same few-shot adaptation sets. our focus is on (a) evaluating in an extreme few-shot learning scenario where no auxiliary data is available (using deepslp), and (b) evaluating in a high-resource setting when additional (auxiliary) training data is available (metaslp). we use bert @xcite as our text encoder throughout to facilitate model comparisons. we report deepslp and metaslp reptile (i.e., using reptile as our meta-learning algorithm) in our main table of results (given their effectiveness), and present additional baselines as well as hyperparameters and training details in appendix a.6. we use bert @xcite as our encoder as it is a text encoder that allows us to get passage-level encodings, it is computationally light-weight compared to decoder-based llms such as llama @xcite and gpt (openai, 2024) (and we can carry out full fine-tuning) and, crucially, it does not suffer from data contamination as, in contrast to the more recent llms, it has not been pre-trained on our test data foot_2 . ## results and discussion due to space restrictions, we present and discuss our best models in ## conclusion and future work we presented a novel few-shot learning paradigm that is based on soft-label prototypes capturing the simultaneous membership of data points over several classes, and demonstrated its effectiveness in low and high-resource settings. we evaluated our approach on 48 different tasks / settings and showed that it outperforms a range of strong baselines. in the future, we plan to use meta-learning algorithms such as pacmaml @xcite and bayesian maml @xcite that relax assumptions with respect to train-test set distributions and thus alleviate this current limitation in our work.