Skill-Based Few-Shot Selection for In-Context Learning (2023)

Abstract: *In-context learning* is the paradigm that adapts large language models to downstream tasks by providing a few examples. *Few-shot selection*—selecting appropriate examples for each test instance separately—is important for in-context learning. In this paper, we propose **Skill-KNN**, a skill-based few-shot selection method for in-context learning. The key advantages of Skill-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself. Technically, Skill-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant surface features. Experimental results across five cross-domain semantic parsing datasets and six backbone models show that Skill-KNN significantly outperforms existing methods.

## introduction in-context learning @xcite has become a prevailing paradigm for utilizing large language models (llms) @xcite @xcite @xcite @xcite @xcite . it employs a frozen taskagnostic backbone model to serve various downstream tasks without requiring parameter updates for each task. under in-context learning, the llms generate output for an input query by conditioning on the prompt that contains input-output examples. due to limited context length of the language model, only a few examples can be presented in the prompt. prior studies have found that the performance of in-context learning is sensitive to the selected in-context examples @xcite @xcite . therefore, one essential research question is: how to select proper examples from a large example bank? raw-input-based selection is one widely applied solution @xcite @xcite . it involves embedding raw inputs of examples using an off-the-shelf embedding model and then selecting the most similar examples. it can be conveniently applied in various downstream tasks. however, this method can be easily biased by surface natural language features that are not important for the target task. for instance, in semantic parsing tasks 1 , the raw-input-based selection just finds out examples with similar entities (as illustrated in figure 1a ), while the better in-context examples should contain the required executable operations in logical forms, which can be regarded as the task-specific skills. to overcome this limitation, we aim to make this embedding-based selection better aware of the intrinsic skills behind the raw inputs. we consider to harness the power of prompting llms to convert the desired skills from raw inputs, which maintains the training-free advantage during selection. there has been much work trying to fine-tune the embedding model for each task based on the example bank @xcite @xcite . however, fine-tuning-based methods are difficult to apply in practical scenarios: it is laborious to train and save the embedding model for each task, and it is also inconvenient to re-train the model on a dynamic example bank that can be updated frequently. specifically, we introduce skill-knn, a training-free, skill-based selection method (briefly illustrated in figure 1b ). overall, skill-knn will first generate skill-based descriptions from raw input queries, then feed these descriptions into an off-the-shelf embedding model to select most similar examples. to generate skill-based descriptions, we prompt a frozen llm with just a few human-1 semantic parsing means to parse an nl utterance into a machine-understandable logical form (e.g., a sql query). the experimental results show that skill-knn brings a considerable boost for in-context learning compared to the raw-input-based selection. we evaluate skill-knn on five challenging semantic parsing datasets: spider @xcite , dr. spider @xcite , kaggledbqa @xcite , bird @xcite , and cogs @xcite . we take six models for in-context learning: text-chat-davinci-002, codedavinci-002, text-davinci-003, code-cushman-002, gpt-35-turbo, and gpt-4. across these tasks and models, skill-knn consistently performs best among non-oracle selection methods and, at times, is even comparable to oracle methods. for instance, with text-chat-davinci-002, skill-knn achieves 78.3% execution accuracy on spider, while the best raw-input-based selection method reaches 74.6% and target-knn foot_0 attains 78.6%. furthermore, our ablation study indicates that skill-knn retains its superiority when constraints are imposed on the annotated demonstrations, including reducing the number of demonstrations, restricting the database diversity, and decreasing the operation coverage. our contributions are three-fold: 1) we propose a skill-based few-shot selection method skill-knn, which leverages the power of prompting ## preliminaries in this section, we introduce embedding-based fewshot selection as the preliminary of our method. ## skill-knn skill-knn involves a rewrite-then-retrieve process to better exploit the potential of in-context learning. figure 2 gives a bird's-eye view of our method. to mine and utilize task-specific skills, skill-knn contains a prompting-based rewriting stage and a skill-based selection stage. prompting-based rewriting will prompt llms to generate skill-based descriptions from the given input query. skill-based selection will return fewshot examples based on these generated descriptions. in the following, we elaborate the design of skill-knn. ## analysis the most important mechanism in skill-knn is the prompting-based rewriting which requires a few manually annotated demonstrations. here we investigate how would these demonstrations affect the performance of skill-knn. we experimentally analyze two factors: the number of annotated demonstrations and the selection of examples for annotation. the following analysis takes the base version of skill-knn with sbert as the embedding model and the spider dataset for evaluation. ## conclusion this work proposes skill-knn to facilitate incontext learning on semantic parsing tasks. by generating skill-based descriptions without any finetuning, skill-knn and its two variant outperform raw-input-based selections in various tasks.