Designing Informative Metrics for Few-Shot Example Selection (2024)

Abstract: Pretrained language models (PLMs) have shown remarkable few-shot learning capabilities when provided with properly formatted examples. However, selecting the “best” examples remains an open challenge. We propose a complexity-based prompt selection approach for sequence tagging tasks. This approach avoids the training of a dedicated model for selection of examples, and instead uses certain metrics to align the syntactico-semantic complexity of test sentences and examples. We use both sentence- and word-level metrics to match the complexity of examples to the (test) sentence being considered. Our results demonstrate that our approach extracts greater performance from PLMs: it achieves state-of-the-art performance on few-shot NER, achieving a 5% absolute improvement in F1 score on the CoNLL2003 dataset for GPT-4. We also see large gains of upto 28.85 points (F1/Acc.) in smaller models like GPT-j-6B.

## introduction pretrained language models (plms) have demonstrated impressive few-shot learning capabilities when prompted with examples for a particular task @xcite . in few-shot prompting, k examples that demonstrate the goal of the downstream task along with the test sample (collectively called the k-shot prompt) are provided to the plm. however, the effectiveness of plms in few-shot learning depends on the "quality" of these prompts; achieving this quality is challenging due to the need for unambiguous instruction @xcite , which is further complicated by the task of selecting appropriate examples. appropriately selected examples will ensure that the plm can generalize effectively and perform well across diverse natural language processing (nlp) tasks. recent work has shown that specific examples provided in-context can significantly impact the performance of plms @xcite . @xcite notes that specific example combinations lead to better generalization, and connects this with the informativeness of these examples. this suggests that surfacing "specific" properties of prompts result in improved performance. most prior work for prompt selection has either trained a prompt retriever to select examples, or used plms themselves to label examples @xcite to serve as a good prompt. however, both approaches require additional computation and task specific training. @xcite use nearest neighbours algorithms to select examples. however, this approach solely relies on embedding space properties, and does not consider other metrics which model label distribution such as entropy, leading to suboptimal performance in some tasks. in this work, we propose a new approach for example selection for sequence tagging tasks. these tasks are important because they enable the extraction of structured information from unstructured text data; this serves as a fundamental building block for various nlp tasks. our approach, complexity-based prompt retrieval (cp retrieval), aims to select examples from the training data whose "properties" align with that of the test sentence. to this end, we use sentence-and word-level complexity measures to match prompt examples to evaluation sentences. we conduct our experiments on named entity recognition (ner), part-of-speech (pos) tagging, and sentence chunking. we also create a new dataset involving tagging of contextual integrity (ci) parameters for privacy policies. we show that cp retrieval improves accuracy significantly in all cases. the contributions of this work are as follows: (1) we propose cp retrieval to select prompts based on sentence and word level complexity metrics ( § 3); and (2) we demonstrate improved few-shot accuracy with cp retrieval. our approach achieves state-of-the-art performance in few-shot ner with a 5% absolute improvement in f1 score on the conll2003 dataset for gpt-4 and upto 28.85 points (f1/accuracy) in smaller models ( § 5). ## related work to perform our experiments, we use the structured prompting paradigm introduced by @xcite as our baseline for named entity recognition (ner), part-of-speech (pos) tagging, and sentence chunking. these are sequence tagging tasksthose that assign labels or tags to tokens or other units of text. this involves iteratively feeding the plm a word and its predicted label to get it to label the next word. plms are also provided k pairs of (sentence, tagged sentence) as the task examples in a k-shot prompt. experiments by @xcite show that plms can effectively solve these tasks with just a few demonstrations using this technique. we consider another task involving annotating contextual integrity (ci) parameters in privacy policies @xcite ; we use a generic few-shot prompt as the baseline for this task as shown in appendix a (fig. 6 ). liu et al. (2022) have explored strategies for selecting examples to leverage gpt-3's few-shot learning capabilities. they analyzed the sensitivity of gpt-3's performance to randomly sampled examples and found significant variability (a phenomenon also explained by @xcite ). @xcite proposed a retrieval based approach called kate that selects semantically similar examples by using k-nearest neighbors (in the embedding space of pre-trained sentence encoders like bert and roberta) which are used as the examples for the k-shot prompt. they found that kate substantially improved gpt-3's results over random sampling on natural language understanding and generation tasks. ## methodology our core idea is simple: to select examples that closely resemble the test sample in both semantics and length while ensuring a diverse range of labels for the task. we measure the "complexity" of a candidate example using 3 different sentenceand word-level metrics: (a) normalized sentence similarity score, (b) normalized smoothed length similarity, and (c) normalized label entropy. problem setup: assume we have a training dataset of the form @xmath0, y i )} n @xmath1, our goal is to use a complexity score to select a subset of examples s of the training dataset (s ⊂ d) s.t. |s| = k. ## experimental setup tasks: we perform our experiments on 3 different sequence tagging tasks namely ner, pos tagging, and sentence chunking. each task involves the annotation of tokens in a sequence with specific labels, facilitating the extraction of valuable information and enhancing language understanding. for all datasets, we use a random set of 1000 test samples. for the ci task, we use 100 test samples. ## results we observe substantial gains using our method, with the largest gains on gpt-j-6b which achieves a 28.85 point increase on ner, followed by gpt-neo-2.7b which achieves a 24.82 point increase in chunking, and finally with gpt-neo-125m which achieves a 15.56 point increase in pos tagci tagging (acc.) ## conclusions in summary, cp retrieval is a flexible technique that can be used to enhance the accuracy of existing few-shot approaches using example scoring. additionally, it provides a task-agnostic quantification of the informativeness of examples for improved model performance. the relative weights of the components of the complexity score can be tuned on a per-task basis. our approach allows us to extract the most out plms without any fine-tuning of model parameters.