Feature-Adaptive and Data-Scalable In-Context Learning (2024)

Abstract: In-context learning (ICL), which promotes inference with several demonstrations, has become a widespread paradigm to stimulate LLM capabilities for downstream tasks. Due to context length constraints, it cannot be further improved in spite of more training data, and general features directly from LLMs in ICL are not adaptive to the specific downstream task. In this paper, we propose a feature-adaptive and data-scalable in-context learning framework (FADS-ICL), which can leverage task-adaptive features to promote inference on the downstream task, with the supervision of beyond-context samples.Specifically, it first extracts general features of beyond-context samples via the LLM with ICL input form one by one, and introduces a task-specific modulator to perform feature refinement and prediction after fitting a specific downstream task. We conduct extensive experiments on FADS-ICL under varying data settings (4~128 shots) and LLM scale (0.8~70B) settings. Experimental results show that FADS-ICL consistently outperforms previous state-of-the-art methods by a significant margin under all settings, verifying the effectiveness and superiority of FADS-ICL. For example, under the 1.5B and 32 shots setting, FADS-ICL can achieve +14.3 average accuracy from feature adaptation over vanilla ICL on 10 datasets, with +6.2 average accuracy over the previous state-of-the-art method, and the performance can further improve with increasing training data.

## introduction in recent years, increasingly large-scale pre-trained language models (llms) have emerged, which leads to a substantial cost for fine-tuning different models for each downstream task. alternatively, in-context learning (icl) has shown impressive performance on nlp downstream tasks without any * corresponding author: zhendong mao. modifications to llms. specifically, icl prepends several input-label pairs (demonstrations) to a test sample via a task-specific template and conducts prediction conditioned on this prompt, namely context, for better inference @xcite . most previous works focus on how to design the best prompt to steer its best performance. for example, some works aim to search for the most suitable templates for specific tasks @xcite , while others focus on selecting the best demonstrations for each test sample @xcite @xcite . there are even a few works that explore the order of demonstrations @xcite . however, due to the context length constraints of llms, these icl-based methods can only accept a limited number of samples as demonstrations put into the context, even if more labeled samples are available, which severely hinders performance on downstream tasks. thus, what we need is data scalability, that is, the model can accept scalable labeled data to enhance model performance. recently, a few works have explored exploiting beyond-context samples, similar to knn-lm @xcite . instead of putting all labeled samples into the context at once, they divide the inference process into two parts: sequentially obtaining the feature representation of each sample, and conducting prediction based on the relevance of the feature representation. specifically, they compute nearest neighbors for the test sample based on the distance in the feature space and adjust prediction results accordingly. for example, knn-prompt @xcite performs the final prediction by interpolating the original predicted distribution with the distribution of nearby samples, while knn-prompting @xcite adopts voting by them directly. however, they ignore feature refinement for specific tasks and instead directly use general features obtained from the llm designed for language modeling, which we refer to as figure 1 : the effect of data scalability and feature adaptation on icl. for data scalability, the performance of vanilla icl cannot be further improved when exceeding a certain amount of data, but knn-prompting and fads-icl can. for feature adaptation, fads-icl conducts feature refinement for specific tasks, so that it outperforms knn-prompting using general features by large margins under all data settings. the issue of feature adaptation. such non-adaptive feature usage can also severely impair the performance on downstream tasks. from figure 1 , we can see that under the framework of icl, data scalability and feature adaptation are both essential to performance on downstream tasks. to this end, we proposed a feature-adaptive and data-scalable in-context learning framework (fads-icl), which can leverage task-adaptive features to promote inference on the downstream task, with the supervision of beyond-context samples. specifically, fads-icl first extracts the general features of each sample by feeding them in the form of vanilla icl to an llm in turn. then, these features and corresponding labels are used to supervise the training of a lightweight modulator for feature adaptation in the downstream task. finally, during inference, this modulator can refine the general features to the task-adaptive features for test samples and perform the final prediction based on them. here, fads-icl solves the issue of data scalability by feeding them in the form of icl to the llm one by one to extract general features, which does not need to put all samples into context. besides, fads-icl also implements feature adaptation by further refining general features through a modulator for a specific downstream task. to verify the effectiveness of fads-icl, we conduct evaluation experiments on 10 established datasets under varying amounts of labeled sam-ples and llm scales. the main experimental results show that fads-icl consistently outperforms state-of-the-art methods by a large margin under all data settings (4∼128 shots, refer to figure 1 ) and all llm scale settings (0.8b∼70b). specifically, under the fixed 1.5b llm and 32 shots, fads-icl can achieve +14.3 average accuracy from feature adaptation over vanilla icl on 10 datasets, with +6.2 average accuracy over the previous state-of-the-art method, knn-prompting. when growing to 128 shots, the improvement over vanilla icl further increases to +17.8, showing its data scalability. in short, feature adaptation can improve downstream performance with a fixed amount of available data, while data scalability comes into play as more data becomes available. further, we systematically analyze the impact of three key settings in fads-icl, including modulators, selection of general features, and the number of demonstrations, which provides in-depth insights and the corresponding recommended configurations in fads-icl. the analysis experiment on different modulators reveals the importance of a parametric modulator that can promote feature adaptation in fads-icl. besides, the analysis experiment on the selection of general features shows the hidden states containing rich semantics are more suitable for fads-icl than probability distribution. further, our study on the role of demonstrations shows that demonstrations during feature extraction are vital to fads-icl while fads-icl is robust to the number of demonstrations. our contributions are summarized below: (1) we propose a feature-adaptive and data-scalable in-context learning framework, which can leverage task-adaptive features to promote inference on the downstream task, with the supervision of beyondcontext samples. to our knowledge, this is the first time that feature adaptation has been considered in icl. (2) extensive experiments are conducted on 10 datasets under varying settings and show that fads-icl consistently outperforms previous state-of-the-art methods by a significant margin in almost all settings. (3) detailed analytical experiments on key settings provide in-depth insights and recommended configurations in fads-icl. ## fads-icl for the sake of achieving both feature adaptation and data scalability, we propose fads-icl. in this section, we elaborate on the architecture of fads-icl, including a feature extractor and a lightweight task-specific modulator. the overall framework is depicted in figure 2 . note that the train set in fads-icl is split to a small demonstration set @xmath0, y s i )} and a residual set @xmath1, y r i )}, with @xmath2. ## analysis in this section, we first systematically analyze the impacts of three key settings in fads-icl including modulators, selection of general features, and the number of demonstrations, which provides indepth insights and the corresponding recommended configurations in fads-icl. visualization for feature adaptation is presented at last. ## related works there are many works contributing to improving the performance of icl on downstream tasks. first, most related works focus on designing the best prefix for each test sample, including demonstration selection, corresponding order, and task-specific templates. specifically, @xcite retrieves demonstration sets based on semantic similarity with an off-shelf sentence encoder, while @xcite ; @xcite train a prompt retriever with weak supervised learning. some works @xcite explore the order of demonstrations, while other works @xcite aim to search for the most suitable templates for specific tasks. these works do not conflict with fads-icl, and you can expect better results when these methods are applied to our framework. second, a few works further finetune the llm based on the input-output format of icl with a large set of tasks, to adapt to icl during inference, e.g. metaicl @xcite , ict @xcite . this type of approach has not become a mainstream application paradigm due to the high cost of finetuning llm. third, some works @xcite have explored exploiting beyondcontext samples, similar to knn-lm @xcite . however, they ignore feature refinement for specific tasks and instead directly use features designed for language modeling in llm, while fads-icl implements feature adaptation, resulting in better performance on downstream tasks. ## conclusion this paper proposes fads-icl, which can leverage task-adaptive features to promote inference on the downstream task with the supervision of beyond-context samples. it implements data scalability by extracting general features using the llm with icl input form one by one, while a lightweight task-specific modulator is introduced to complete feature refinement and final prediction, achieving feature adaptation. fads-icl consistently outperforms previous state-of-the-art by a significant margin under all data and llm scale settings, verifying its effectiveness and superior-ity. detailed analytical experiments on key settings provide more in-depth insights and recommended configurations in fads-icl.