k NN P ROMPTING : B EYOND -C ONTEXT L EARNING WITH C ALIBRATION -F REE N EAREST N EIGHBOR I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
In-Context Learning (ICL), which formulates target tasks as prompt completion conditioned on in-context demonstrations, has become the prevailing and standard utilization of LLMs.
In this paper, we ﬁrst disclose an actual predicament for this typical usage that it can not scale up with training data due to context length restriction.
Besides, existing works have shown that ICL also suffers from various biases and requires delicate calibration treatment. To address both challenges, we advocate a simple and effective solution, k NN Prompting, which ﬁrst queries LLM with training data for distributed representations, then predicts test instances by simply referring to nearest neighbors. We conduct comprehensive experiments to demonstrate its two-fold superiority: 1) Calibration-Free : k NN
Prompting does not directly align LLM output distribution with task-speciﬁc label space, instead leverages such distribution to align test and training instances.
It signiﬁcantly outperforms state-of-the-art calibration-based methods under comparable few-shot scenario.
2) Beyond-Context : k NN Prompting can further scale up effectively with as many training data as are available, continually bringing substantial improvements. The scaling trend holds across 10 orders of magnitude ranging from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B to 30B. It successfully bridges data scaling into model scaling, and brings new potentials for the gradient-free paradigm of LLM deployment.
I NTRODUCTION
[Figure] Large language models (LLMs), when scale up to billions of parameters, have demonstrated remarkable capabilities in a wide range of NLP tasks (Radford et al., 2019; Brown et al., 2020).
However, such models are prohibitively expensive to train with most of the research- or consumer-level devices, though some of them are already publicly available (Zhang et al., 2022).
As a result, it is now an emerging paradigm that LLMs are hosted in a remote data center while accessed by end users or applications via simple API requests 1 .
The typical usage of LLM under such paradigm is InContext Learning, where LLM reads and completes a prompt sequence as how it is pretrained on massive text corpora.
The prompt is constructed by concatenation of several training examples and a test instance, and the prediction is obtained by mapping the LLM word continuations back to label space.
It is widely investigated and acknowledged that modern neural networks generally perform better w.r.t.
increased training data.
Speciﬁcally, there exists a power law between expected model performance and available data scale (Hestness et al., 2017; Rosenfeld et al., 2020).
For ICL, it is also empirically observed that the performance continually improves when more training examples are prepended into the prompt (Brown et al., 2020).
However, such improvements are quickly prevented by the predicament of context length restriction, as language models are designed and trained to only process sequences within a ﬁxed length, which is in fact 1024 or 2048 tokens.
In order to utilize more training data, several works try to select the most relevant examples to compose the prompt before querying LLM (Liu et al., 2022b; Rubin et al., 2022), but still only in-context examples can actually participate the LLM inference while most training data are discarded beforehand, thus providing marginal data scaling beneﬁts. Besides, their reliance on external retriever also incurs further complications.
As a consequence, such a situation poses a serious challenge for many practical scenarios where more than a few training data are available. Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation (Holtzman et al., 2021) and instability (Lu et al., 2022) as shown in existing works. Accordingly, many have proposed various ways to calibrate the output distribution (Zhao et al., 2021; Jiang et al., 2021; Min et al., 2022a).
For example, Zhao et al.
(2021) measure such bias by probing LLM with a ” NA ” example and record the according prior. However, as LLMs are pretrained on general-domain natural language, its capability to complete a fabricated prompt is essentially not aligned with downstream task-speciﬁc label space. As a consequence, such calibration-based methods can only alleviate the bias to a limited extent. In this paper, we advocate a simple and effective solution, k NN Prompting, to address both challenges.
Speciﬁcally, we assign training data into a demonstration set and an anchor set.
We append each anchor example into the prompt and query LLM, then instead of aligning word continuations with labels, we collect the language modeling probability as distributed representation and cache it into a local datastore.
At inference time, for each test instance, we similarly obtain its representation and match it against the maintained datastore to make predictions.
In general, the proposed framework enables both calibration-free optimization because it avoids forced input-label alignment, and beyond-context learning because the anchor set allows utilization of unlimited training data. We conduct comprehensive experiments using 10 established text classiﬁcation tasks to demonstrate the signiﬁcant superiority of k NN Prompting across various scenarios and against competitive opponents: 1) Under few shot scenario where training data is very limited and ﬁts in the context, k NN Prompting outperforms state-of-the-art calibration-based methods by considerable margin (up to +7.07).
2) Under low resource or fully supervised scenario where training data can not ﬁt in the context, k NN Prompting further exhibits its major advantage.
It can effectively scale up with as many training data as are available across 10 orders of magnitude (2 shots ∼ 1024 shots, see Figurefor illustration) as well as different LLMs scales (0.8B ∼ 30B). Speciﬁcally, with only 32 shots training data, it dramatically improves ICL by +13.58 in average score at its most, and achieves absolute improvements up to +18.84 under fully supervised setting.
We also provide formal explanation on the intrinsic mechanism of effectiveness, as well as detailed analysis regarding its robustness, efﬁciency and choices of design.
Accompanied with these appealing aspects, k NN Prompting is in general a promising solution that bridges the beneﬁts of data scaling into model scaling to take the gradient-free paradigm of LLM deployment one step further.
B ACKGROUND : I N -C ONTEXT L EARNING
In this section, we formulate the task and recap the ICL baseline.
Assuming a target task with training data set T =
( x i , y i ) } , and Y as its categorical label space.
At inference time, the model is asked to predict y test given test instance x test .
We then denote an LLM θ that is pretrained with a standard language modeling objective.
At employment, it samely predicts a probability distribution p ( w t | w <t , θ ) for the next token at t -th position conditioned on previous context w <t .
In-context learning ﬁrst formulates training examples { ( x i , y i ) } in the format of input-label mapping via intuitive templates (see Appendix F for illustration), and concatenates them into a natural language sequence along with the test instance to construct a prompt:
[Equation]
where π denotes template-based transformation (see Appendix for illustration) and ⊕ denotes concatenation operation.
Note that π implies a verbalization process that maps label space Y to cor- [Table] responding tokens V picked from the LM vocabulary.
When queried by the prompt P , LLM will try to mimic the prepended training examples in the context and predict a probability distribution p ( v | P, θ ) for the next token v .
We then map it back to label space Y as prediction:
[Equation]
We ﬁrst conduct a pilot study to verify its performance with increased training examples.
Figure 2 shows that when prompt P includes more available training data as demonstrations, the performance consistently improves, starting from one shot to the maximum allowed number of examples for each task.
This is in accord with the power law of data scaling investigated in many existing studies (Hestness et al., 2017; Rosenfeld et al., 2020).
However, this trend is then prevented by the context length restriction, stopped at 8 shots for SUBJ dataset, and 16 shots for SST2.
We provide more comprehensive statistics in Table 1 and Appendix A.
Although very few LM (Beltagy et al., 2020) supports even longer context up to 4096 tokens, the limitation is ultimately not removed.
In conclusion, this situation poses an actual challenge in many scenarios where one would further collect training examples from fewshots to dozens and expect improved performance, but the power law fails.
k NN P ROMPTING
In this section, we introduce the k NN Prompting framework.
For a training data set T , we split and exploit it in two respective usage: T =
D ∪ A , i.e., a demonstration set D
= { ( x di , y di ) } and an anchor set A
= { ( x ai , y ai ) } .
k NN Prompting consists of two stages namely meta test and formal test, the overall framework is illustrated in Figure 3.
Meta Test We build a datastore that caches all anchor examples in A for later inference time usage.
For each x ai , we respectively concatenate it into prompt P , where the prompt preﬁx is constructed using the demonstration set the same as Equation 1:
[Equation]
By querying LLM using P i , we obtain the distribution p ( v | P i , θ ) .
Instead of mapping it back to label space V , we cache the entire language modeling distribution as the key representation:
[Equation]
Accordingly, label y ai is the value.
The entire datastore thus consists of paired { k i , y ai } , we denote the set of keys as K .
Formal Test At inference time, for each test instance x test , we construct the same prompt as
Equation 1, and obtain distribution p test = p ( v | P test , θ ) .
We then match the distribution against cached K in the datastore, where standard KL divergence is used to measure the distance:
[Equation]
Meta-Test
Formal-Test
LLM
Interface 𝑃 (𝑥 ! !
,∗) (𝑥 ! " ,∗) … (𝑥 ! 𝒜 ,∗) (𝑥 "#$" ,∗) Datastore 𝑘 Nearest Neighbors
KL Divergence 𝒌 ! !
#! 𝑦
# "#$" 𝒌 ! !
𝒌 ! "
𝒌 ! # 𝒌 !
𝒜 (𝑥 % ! ,𝑦 % ! ) …
(𝑥 % ! ,𝑦 % ! )
(𝑥 % |𝒟| ,𝑦 % |𝒟| ) Demonstration
Anchor Set
Distribution
Key-Distance Value 𝒌 ! ! :
0.136 𝒌 ! " :
0.227 𝒌 !
… … 𝑦 ! !
𝑦 ! "
𝑦 !
The predictions are calculated by aggregating its k nearest neighbors:
[Equation]
where NN k ( ∗ , K ) denotes the set of k nearest neighbors in K .
E XPERIMENTS 4. S ETUP
Dataset and LLM
We use 10 established text classiﬁcation datasets, respectively SST2 (Socher et al., 2013), SUBJ (Pang & Lee, 2004), MPQA (Wiebe et al., 2005), AGNews (Zhang et al., 2015),
CB (De Marneffe et al., 2019), CR (Hu & Liu, 2004), DBPedia (Zhang et al., 2015), MR (Pang &
Lee, 2005), RTE (Dagan et al., 2005) and TREC (Voorhees & Tice, 2000).
For each dataset, we devise intuitive template for ICL (Appendix F), and other regarding statistics are listed in Appendix A.
We employ the most prevalent and extensively benchmarked GPT series model for experiments.
We investigate a wide range of LLM scales, including GPT2 Large (0.8B), XL (1.5B)
(Radford et al., 2019) and the OPT (Zhang et al., 2022) series (3B-30B) which is accessible and comparable to
GPT3 (Brown et al., 2020).
GPT2 XL is used for most analyses unless explicitly indicated.
Implementation We invariantly set the number of neighbors k to 3.
There are no other hyperparameters as the entire framework is training-free.
We run with 5 different random seeds for
Figure 5 and Figure 6, and 10 random seeds for all other results.
Mean and standard deviation are reported for robust conclusions.
The largest 30B model requires 2 × 40GB memory for inference with model parallelism and FP16 mode.
Our code will be available at https: //github.com/anonymous/knnprompting .
4. D ATA U TILITY
In this paper, we refer to data utility as whether and how performance scales up with increased training data.
It can be formally depicted by the power law (Hestness et al., 2017):
[Equation]
where m is the training data size, α is a constant scaling factor, β describes the exponential steepness, and ε ( ∗ ) calculates the generalization error, i.e., test time performance.
We refer m to training shots, i.e., number of examples per class throughout the paper.
In the following sections, we investigate various settings of m , respectively few shot ( m ≤ M T ), low resource ( m = 128 ), and the overall scaling law ( m ≤ 1024 ).
We refer M T to the maximum training shots allowed in context for each task T , the speciﬁc statistics can be found in appendix A.
4.2. D ATA U TILITY U NDER F EW S HOT S CENARIO
We ﬁrst investigate k NN Prompting under few shot setting, i.e., m ≤ M T .
We simply set |D| =and use all other examples for A .
We leave further exploration of split strategies to Appendix B.3. In
[Table] order to avoid context length restriction and maintain comparability to baselines as much as possible, 6 out of 10 tasks where M T ≥ 8 are selected. The baselines are ICL and state-of-the-art calibrationbased augmentations including Contextual Calibration (Zhao et al., 2021) and Noisy Channel (Min et al., 2022a).
Contextual Calibration probs the prior bias w.r.t.
each category and accordingly calibrate the outputs.
Noisy Channel formulates ICL as computing input likelihood conditioned on labels. Results in Table 2 show that k NN Prompting signiﬁcantly outperforms competitive baselines under strictly comparable settings ( m ≤ 8 ), speciﬁcally, +3.56 for 4 shot, and +7.07 for 8 shot. As a calibration-free method, k NN Prompting demonstrates clear superiority.
4.2. D ATA U TILITY B EYOND THE C ONTEXT
We then investigate a major advantage of k NN Prompting, which is scaling up to more training examples that otherwise would not ﬁt in the context.
We increase m to 128 and compare with: 1)
ICL Ensemble which is an intuitive alternative to scale ICL up, and has been adopted in previous works (Jiang et al., 2020); 2) ﬁnetuning of standard PLM such as BERT or GPT Large, which could produce meaningful results with such amount of data.
For all methods, we append maximum M T examples into prompt P .
For ICL Ensemble, we split T into multiple non-overlap demonstration sets T
= D 1 ∪ D 2 ∪ ... ∪ D N to construct different prompts, and ensemble their predictions.
Results in Table 3 show that k NN Prompting continues to improve and outperform ICL and its ensemble respectively by +16.96 and +16.08 (0.8B model, average score).
Besides, the ensemble baseline is also very inefﬁcient, assuming M T = 8 , we need to query 128 / 8 = 16 times for every test instance, and this keeps growing linearly if we use more training data, eventually becomes prohibitively inefﬁcient and can not scale up either. k NN Prompting also outperforms FT when the adopted LLM scales above 6B (82.73). By contrast, it would require LLM to scale above 30B (82.45) using ICL Ensemble and even larger using standard ICL.
4.2. C ONTINUALLY S CALING U P TO T HOUSANDS OF T RAINING D ATA
We now fully scale data up to thousands of training examples and provide extensive results across model scales to observe their overall scaling performance.
Figure 5 shows k NN prompting can continually generalize across the tested range to provide effective data utility, re-enabling the power law [Table] Compared with calibration-based methods. under gradient-free paradigm of LLM deployment.
The full results can be found in Appendix C.2.
With only 32 shots training data, k NN prompting dramatically improves ICL by +13.58 in average score at its most (0.8 B), and achieves absolute improvements up to +18.84 under fully supervised setting. With the largest model OPT 30B, it achieves a best performance of 86.02.
Comparison to PromptCompose A line of related works try to utilize available training data by ﬁrstly retrieving the most relevant ones from the entire training set, then selectively composing the prompt before querying LLM (Liu et al., 2022b; Rubin et al., 2022). We refer to such methods as PromptCompose, and reproduce according to Liu et al. (2022b).
We employ state-of-the-art generalpurpose sentence encoders 4 to represent test and training instance, and compute their cosine similarity, the most similar M T examples are selected to construct prompt P . These retrieving models include BM25 (Trotman et al., 2014), Sentence-BERT (Reimers & Gurevych, 2019), SimCSE (Gao et al., 2021) and Trans-Encoder (Liu et al., 2022a). Figure 6 shows that although such methods indeed exhibits marginal scaling beneﬁts, they are nowhere near competitive against k NN Prompting. Full results are listed in Appendix C.3.
To further solidify this conclusion, we push PromptCompose to an extreme situation trying to approximate its upper-bound. As such methods ultimately resort to compose the prompt, it should be bounded by the best composition scheme from ﬁnite compositions. We thus search for 1,000 prompts with different examples and report their best run.
In Table 4 we ﬁnd k NN Prompting performs on par with or even surpasses such upper-bound approximation.
In conclusion, k NN Prompting essentially [Table] makes better use of training examples than PromptCompose as the latter still only refer to in-context examples during LLM inference while most training data are discarded beforehand.
4. A NALYSES AND E XPLANATION 4.3. R OBUSTNESS W . R . T . D IFFERENT T RAINING E XAMPLES
It is previously found that vanilla ICL suffers from severe instability (Zhao et al., 2021). Figureis produced with 10 different seeds, which results in different choices and permutations of training data. We show that k NN Prompting signiﬁcantly improves the robustness. Besides, the performance becomes more robust with increasing anchor set size.
On 10 investigated tasks, the standard deviation of k NN Prompting (3.83) is signiﬁcantly smaller than ICL (9.14) 5 .
4.3. R OBUSTNESS U NDER I MBALANCED S CENARIO
We further test the robustness of k NN Prompting under imbalanced label scenario. Take binary classiﬁcation like SST2 as example, we simulate imbalance ratio by controlling one of the category proportionally to { 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625 } of the entire training set, where 0.5 corresponds to the ordinary balanced scenario. We keep the test set intact, which results in a challenging out-of-distribution (OOD) setting.
Results in Figure 8 reveal the vulnerability of the proposed method.
Under imbalanced setting, k NN Prompting is overwhelmed by the large quantity of anchors from the majority class where it is simply far more easier to ﬁnd closer neighbors. To address such performance degradation under challenging imbalanced scenario, we propose a simple normalization trick: we average the anchor representations to produce one centered anchor for each class, the resulting anchor is thus more representative and also avoids quantity distraction.
Such an adaptation works surprisingly well with no loss of performance even under ordinary balanced setting. Similar conclusion can also be drawn on more tasks (see Appendix C.4).
ICL Pred 𝑘 NN Prompting
Acc [Figure] [Figure]
film ( ✘ ) [Figure] ✘ )
[Figure] ✘ ) [Figure] 4.3. Q UALITATIVE A NALYSES AND R EASONS OF E FFECTIVENESS
We ﬁrst formally organize the explanation as follows according to Figure 9:
• The output language modeling (LM) distribution of LLM is essentially not well aligned with taskspeciﬁc label space, resulting in inferior performance ( 83.5 test accuracy ) of default ICL.
• If we similarly perform inference on anchor set, we would expect to get approximately 83.5 anchor accuracy by assuming i.i.d. data distribution. However, we are already aware of each of their golden labels, which actually gives 100 anchor accuracy .
• LM distribution is inferior for making direct predictions, but superior for matching examples because it entails distributional, delicate and comprehensive representations generated by LLM. k
NN Prompting leverages such representations ( 83.5 accuracy ) only for matching, and refer to their golden labels ( 100 accuracy ) for predicting, thus successfully transfer part of the knowledge originating from anchor labels to test instances. In the visualization, the representations generally exhibit partially clustered pattern, we can identify proportional examples that get entangled with different categories and crowded together (Case A, C,
D), these confusing cases are likely to cause erroneous predictions in ICL and corresponds to underperformed 83.5 accuracy as mentioned above.
Speciﬁcally, case A is an abstract about a novelist and should belong to category artist , but it is easily confused with category book using ICL because the context did mention books.
By contrast, k NN Prompting can correctly predict by referring to similar anchors that are also about novelists and their books (as listed in the table).
Some of the anchors are also incorrectly predicted as book , but it no longer matters because k NN Prompting only use the distribution for nearest neighbor search but refer to golden labels for prediction. Besides, as we can clearly know how the prediction is made, i.e., which anchor examples are referred, the proposed method also exhibits explainability as a further advantage.
R ELATED W ORKS
Large language models, since ﬁrstly scaled up to hundreds of billions parameters by Brown et al.
(2020) and followed by several others (Rae et al., 2021; Zhang et al., 2022;
Chowdhery et al., 2022; Cohen et al., 2022; Smith et al., 2022), have become the most prominent direction of NLP area.
Although these models exhibit surprisingly powerful and even emergent capabilities in a wide range of NLP tasks (Wei et al., 2022b; Hendrycks et al., 2020; Srivastava et al., 2022), they are prohibitively expensive for most researchers or users to train or even hold.
In-Context Learning, which suits LLM to various tasks while requires no training, therefore becomes the typical usage as is popularized by Brown et al.
Similar ideas of formulating target tasks into natural language sequences can also be found in earlier works (Trinh & Le, 2018; Raffel et al., 2020).
To better exploit LLM for various scenarios, it becomes a crucial problem to develop further augmented methods built upon ICL.
Xie et al.
(2022) provide theoretical explanations that formalize
ICL as Bayesian inference.
Sun et al.
(2022) formally conceptualize the gradient-free utilization of
LLM as Language-Model-as-a-Service (LMaaS).
Wei et al.
(2022a)
and Sanh et al.
propose instruction tuning, which further pretrains LLM with a collection of downstream tasks in a shared prompting format.
Min et al.
(2022b)
and
Chen et al.
(2022b) introduce meta-learning to better adapt LMs to ICL.
Wei et al. (2022c) and Kojima et al.
(2022) propose to augment the demonstrations with human-aided reasoning steps or hints, which surprisingly improved the performance for arithmetics and other reasoning tasks.
Closely related to this work, Liu et al.
(2022b)
and
Rubin et al.
(2022) propose to compose prompt P by selecting most similar training examples. Zhao et al. (2021) and Min et al. (2022a) propose to calibrate ICL prediction via either probing the bias or reversing the conditional prediction formulation.
k NN is a classical machine learning algorithm (Fix & Hodges, 1989) well known for its simplicity and inspired a wide range of application (Papernot & McDaniel, 2018; Orhan, 2018).
In the ﬁeld of NLP, Kaiser et al.
(2017) construct a differentiable memory module for nearest neighbor searching which improves generalization to rare events.
Similar idea has also been explored for generation tasks (Guu et al., 2018), such as dialog generation (Weston et al., 2018), machine translation (Khandelwal et al., 2021), etc.
Wang et al.
and
Chen et al.
(2022a) propose to retrieve similar training examples and incorporate them into the input to jointly train the model.
While Khandelwal et al.
(2020) and Shi et al.
(2022) consider unsupervised corpus as datastore, retrieve and interpolate them with the current step language modeling probability.
The retrieved corpus can also serve as references for knowledge intensive tasks, but the retriever would need explicitly training for such purpose (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022).
Different from these works, k NN Prompting is suitably situated in the gradient-free paradigm of LLM deployment, which avoids calibration treatment and effectively bridges data scaling into model scaling.
D ISCUSSION
Under the existing ICL paradigm, it is often impossible to take advantage of both the capability of
LLM and the data utility of ﬁnetuning, i.e., model scaling and data scaling.
k NN Prompting ﬁnds an effective solution to promise them both.
Nevertheless, we assume its data utility should still be inferior to the specialized ﬁnetuning of LLMs, if given sufﬁcient computation resources in an ideal setting.
We believe that it is a very important and promising direction to further approach this upper-bound and expect to raise more interests in future works.
A potential concern for retrieval-based models is their efﬁciency, especially when corpus level datastore is utilized.
k NN Prompting is free of such concerns as it considers training data, which is in manageable scale.
Under few shot scenario, k NN Prompting even reduces computational costs at deployment time compared to standard ICL. It works well with one shot demonstrations as experimented in Section 4.2.1, while the anchor examples are queried for only once and cached locally. By contrast, existing methods only perform better when all examples are prepended in a single prompt.
This advantage is rather important as we need to repeatedly query the prompt in practical usage of LLM service, and longer prompt results in linearly more monetary costs if charged by token numbers or super-linearly more computational costs if measured by FLOPS.
C ONCLUSION
In this paper, we propose k NN Prompting as a simple and effective solution to advance gradient-free deployment of LLM inference. Motivated as calibration-free optimization, k NN Prompting signiﬁcantly outperforms state-of-the-art calibration-based methods under comparable few shot scenario. While its major advantage is further revealed when training data increases and can not ﬁt in the context.
k NN Prompting can effectively scale up with as many training data as are available, successfully bridging the utility of data scaling into model scaling. We explain the intrinsic mechanism for its effectiveness in the hope of inspiring further insights.
We also analyze other appealing aspects including its robustness and efﬁciency.
The proposed framework endeavors to realize more effective, efﬁcient and applicable utilization of large language models in realistic scenarios, and hopefully could inspire further research interests toward the same goal.
E THICAL C ONSIDERATIONS
This work is built upon the ICL paradigm and involves querying LLM for responses.
These models might generate contents with potential ethical risks regarding fairness and bias (Bommasani et al., 2021;
Blodgett et al., 2020), depending on speciﬁc downstream tasks.
Although the scope of this paper remains on how to better exploit LLM for task performance, it is worth further discussion to combine the proposed framework in conjunction with well-established methods that can measure (Nadeem et al., 2021) and mitigate (Nadeem et al., 2021; Gupta et al., 2022) such ethical risks
A D ATASET S TATISTICS : M AX S HOT I N C ONTEXT
We provide detailed statistics about the number of maximum shots in Table 5, i.e., M T for each task, corresponding to Table 1 in the main manuscript.
[Table]
can allow.
We provide maximum shots under 5% Truncation Probability (TP) restriction as well as the actual M T taken in this paper which is set for
B A BLATIONS B. D ISTANCE M EASUREMENT
We investigate euclidean distance as an alternative distance measurement, which has also been explored in Khandelwal et al. We take the contextual representation h of LLM, and denotes their distance as D L 2 ( h test , h i ) . Table 6 shows that both methods are effective but D KL (Equation 5) based on LM distribution p is a superior measurement.
Actually, p is a projection of h through the word embedding, we think this procedure exploits the well-structured word embeddings of LLM to provide more disentangled representations, thus can better serves as the anchor space.
[Table]
B. E MPIRICAL C HOICE OF k
In Figure 10 we search for different choices of k on MPQA, and found that it is generally a rather robust choice within the wide range [3 , |A| / 2 − 1] .
B. S PLIT S TRATEGY OF D EMONSTRATION AND A NCHOR S ET
For fully supervised scenario where m (cid:29) M T , we can simply set |D| to its maximum M T .
Otherwise, we might need to deliberate the trade-off between the demonstration set and anchor set.
In
Figure 11 we search through all possible combinations given that |A| + |D| ≤ M T .
We see the left part of the heatmap generally performs inferior, i.e., |A| ∈ { 1 , 2 } .
This means a larger |A| contributes more to the performance while the choice for |D| is relatively more robust.
The conclusion also corresponds to few shot results that anchor set yields better data utility than context concatenation.
C C OMPLETE R ESULTS C. D ATA S CALING C URVE U NDER F EW - SHOT S CENARIO
We provide scaling curve w.r t. each speciﬁc dataset in Figure 12, corresponding to Table 2 and
Figure 4 in the main manuscript.
C. D ATA S CALING R ESULTS U NDER F ULLY S UPERVISED S CENARIO
We provide comprehensive results of k NN Prompting across data scales and LLM scales in Table 7, corresponding to Figure 5 in the main manuscript. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
C. F ULL R ESULTS OF C OMPARISON TO P ROMPT C OMPOSE
We provide the full results of comparison between k NN Prompting and PromptCompose in Table 8, corresponding to Figure 6 in the main manuscript.
The employed sentence encoder can be found at https://huggingface.co/models 789 .
[Table]
C. M ORE R ESULTS ON I MBALANCED S CENARIO
We provide evaluation of k NN Prompting under imbalanced scenario on more datasets in Figure 13, corresponding to Section 4.3.2.
We compare k NN Prompting to standard PLM ﬁnetuning in a more extensive data scale.
For ﬁnetuning baselines in both Table 3 and Figure 14, we set hyper-parameters following previous works (Schick & Sch¨utze, 2021).
We set learning rate to 1e-5, batch size to 16, and training steps to 125, 250 or 500, respectively for m ∈ { 32 , 64 } , { 128 , 256 } , { 512 , 1024 } .
For CB, AGNews and RTE, batch size is adjusted to 8, for DBPedia, batch size is adjusted to 4 to avoid OOM.
We observe that with the same model scale, k NN Prompting is superior than ﬁnetuning under the low resource setting, but inferior under fully supervised setting.
This indicates its data utility factor α is still smaller than ﬁnetuning.
However, the main advantage of k NN Prompting comes with LLM, which signiﬁcantly outperforms the ﬁnetuning baseline without any gradient-based optimization.
We have also discussed this in Section 6.
E M ORE C ASE S TUDY
We provide more case study, respectively on SST2 (Figure 15), AGNews (Figure 16), and TREC (Figure 18). 7 https://huggingface.co/sentence- [Figure]
F P ROMPT T EMPLATE
See Table 9 for the used templates (Adopted from Lu et al.
They are intuitively designed and the proposed method should be robust with choices of templates. ICL Pred 𝑘 NN Prompting Acc 81.
B positive ( ✘ ) negative
C positive ( ✘ ) negative
D positive ( ✘ ) negative E positive ( ✘ ) negative A positive ( ✘ ) negative A (Test) looks and feels like a project better suited for the small screen .
ICL Pred
Golden Label distance ↓
[Figure]
NN (Anchor) see `` simone , '' and consider a dvd rental instead positive ( ✘ ) negative 0.
it 's a thin notion positive ( ✘ ) negative 0.
last
[Figure]
minute positive ( ✘ ) negative 0.
ICL Pred 𝑘 NN
Prompting Acc 72. B technology ( ✘ ) business C technology ( ✘ ) business
D technology ( ✘ ) business
E world ( ✘ ) sports
A technology ( ✘ ) business
[Figure]
[Table]
ICL Pred
GoldenLabel distance ↓
[Figure]
[Table]
[Table]
business 0.
[Table]
world ( ✘ ) business 0.
[Table]
[Figure]
business 0. ICL Pred 𝑘 NN Prompting Acc 72.
B positive ( ✘ ) negative
C positive ( ✘ ) negative
D positive ( ✘ ) negative
E positive ( ✘ ) negative
A positive ( ✘ ) negative
A (Test) "it's not a bad premise , just a bad movie ."
ICL Pred
GoldenLabel distance ↓
[Figure]
NN (Anchor) "as it abruptly crosscuts among the five friends , it fails to lend the characters\' individual stories enough dramatic resonance to make us care about them ."
positive ( ✘ ) negative 0.
"the three leads produce adequate performances , but what\'s missing from this material is any depth of feeling ."
positive ( ✘ ) negative 0.
"morrissette has performed a difficult task indeed - he\'s taken one of the world\'s most fascinating stories and made it dull , lifeless , and irritating ."
positive ( ✘ ) negative 0.
ICL Pred 𝑘 NN
Prompting
Acc 54.
B entity ( ✘ ) human
C number ( ✘ ) entity
D number ( ✘ ) expression
E entity ( ✘ ) human
A entity ( ✘ ) description
[Figure]
What is amitriptyline ?
ICL Pred
Golden Label distance ↓
[Figure] NN ( [Figure] What is digitalis ?
human ( ✘ ) description 0.
What is BPH ?
number ( ✘ ) expression 0.
What is agent orange
number ( ✘ ) description 0.
[Table]