k NN P ROMPTING : B EYOND -C ONTEXT L EARNING WITH C ALIBRATION -F REE N EAREST N EIGHBOR I NFERENCE
A BSTRACT
In-Context Learning (ICL), which formulates target tasks as prompt completion conditioned on in-context demonstrations, has become the prevailing utilization of LLMs.
In this paper, we ﬁrst disclose an actual predicament for this typical usage that it can not scale up with training data due to context length restriction.
Besides, existing works have shown that ICL also suffers from various biases and requires delicate calibration treatment.
To address both challenges, we advocate a simple and effective solution, k NN Prompting, which ﬁrst queries LLM with training data for distributed representations, then predicts test instances by simply referring to nearest neighbors.
We conduct comprehensive experiments to demonstrate its two-fold superiority: 1) Calibration-Free : k NN Prompting does not directly align LLM output distribution with task-speciﬁc label space, instead leverages such distribution to align test and training instances.
It signiﬁcantly outperforms state-of-the-art calibration-based methods under comparable few-shot scenario.
2) Beyond-Context : k NN Prompting can further scale up effectively with as many training data as are available, continually bringing substantial improvements.
The scaling trend holds across 10 orders of magnitude ranging from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B to 30B.
It successfully bridges data scaling into model scaling, and brings new potentials for the gradient-free paradigm of LLM deployment.
Code is publicly available 1 .
I NTRODUCTION
[Figure]
Figure brings substantial improvements over standard ICL, and can continually scale up beyond the context with as many data as are available.
Conducted with GPT XL.
Large language models (LLMs), when scale up to billions of parameters, have demonstrated remarkable capabilities in a wide range of NLP tasks (Radford et al., 2019; Brown et al., 2020).
However, such models are prohibitively expensive to train with most of the research- or consumer-level devices, though some of them are already publicly available (Zhang et al., 2022).
As a result, it is now an emerging paradigm that
LLMs are hosted in a remote data center while accessed by end users or applications via simple
API requests 2 .
The typical usage of LLM under such paradigm is In-Context Learning, where
LLM reads and completes a prompt sequence as how it is pretrained on massive text corpora.
The prompt is constructed by concatenation of several training examples and a test instance, and the prediction is obtained by mapping the LLM word continuations back to label space.
It is widely investigated and acknowledged that modern neural networks generally perform better w.r.t.
increased training data.
Speciﬁcally, there exists a power law between expected model performance and available data scale (Hestness et al., 2017; Rosenfeld et al., 2020).
For ICL, it is also empirically observed that the performance continually improves when more training examples are prepended into the prompt (Brown et al., 2020).
However, such improvements are quickly prevented by the predicament of context length restriction, as language models are designed and trained to only process sequences within a ﬁxed length, which is in fact 1024 or 2048 tokens.
In order to utilize more training data, several works try to select the most relevant examples to compose the prompt before querying LLM (Liu et al., 2022b;
Rubin et al., 2022), but still only in-context examples can actually participate the LLM inference while most training data are discarded beforehand, thus providing marginal data scaling beneﬁts.
Besides, their reliance on external retriever also incurs further complications.
As a consequence, such a situation poses a serious challenge for many practical scenarios where more than a few training data are available.
Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation (Holtzman et al., 2021) and instability (Lu et al., 2022) as shown in existing works.
Accordingly, many have proposed various ways to calibrate the output distribution (Zhao et al., 2021; Jiang et al., 2021;
Min et al., 2022a).
For example, Zhao et al.
(2021) measure such bias by probing LLM with a ” NA ” example and record the according prior.
However, as LLMs are pretrained on general-domain natural language, its capability to complete a fabricated prompt is essentially not aligned with downstream task-speciﬁc label space.
As a consequence, such calibration-based methods can only alleviate the bias to a limited extent.
In this paper, we advocate a simple and effective solution, k NN Prompting, to address both challenges.
Speciﬁcally, we assign training data into a demonstration set and an anchor set.
We append each anchor example into the prompt and query LLM, then instead of aligning word continuations with labels, we collect the language modeling probability as distributed representation and cache it into a local datastore.
At inference time, for each test instance, we similarly obtain its representation and match it against the maintained datastore to make predictions.
In general, the proposed framework enables both calibration-free optimization because it avoids forced input-label alignment, and beyond-context learning because the anchor set allows utilization of unlimited training data.
We conduct comprehensive experiments using 10 established text classiﬁcation tasks to demonstrate the signiﬁcant superiority of k NN Prompting across various scenarios and against competitive opponents: 1) Under few shot scenario where training data is very limited and ﬁts in the context, k NN Prompting outperforms state-of-the-art calibration-based methods by considerable margin (up to +7.07).
2) Under low resource or fully supervised scenario where training data can not ﬁt in the context, k NN Prompting further exhibits its major advantage.
It can effectively scale up with as many training data as are available across 10 orders of magnitude (2 shots ∼ 1024 shots, see Figurefor illustration) as well as different LLMs scales (0.8B ∼ 30B).
Speciﬁcally, with only 32 shots training data, it dramatically improves ICL by +13.58 in average score at its most, and achieves absolute improvements up to +18.84 under fully supervised setting.
We also provide formal explanation on the intrinsic mechanism of effectiveness, as well as detailed analyses regarding its robustness and choices of design.
Accompanied with these appealing aspects, k NN Prompting is in general a promising solution that bridges the beneﬁts of data scaling into model scaling to take the gradientfree paradigm of LLM deployment one step further.
B ACKGROUND : I N -C ONTEXT L EARNING
In this section, we formulate the task and recap the ICL baseline.
Assuming a target task with training data set T =
( x i , y i ) } , and Y as its categorical label space.
At inference time, the model is asked to predict y test given test instance x test .
We then denote an LLM θ that is pretrained with a standard language modeling objective.
At employment, it samely predicts a probability distribution p ( w t | w <t , θ ) for the next token at t -th position conditioned on previous context w <t .
In-context learning ﬁrst formulates training examples { ( x i , y i ) } in the format of input-label mapping via intuitive templates (see Appendix F for illustration), and concatenates them into a natural
[Table]
language sequence along with the test instance to construct a prompt:
[Equation]
where π denotes template-based transformation (see Appendix for illustration) and ⊕ denotes concatenation operation.
Note that π implies a verbalization process that maps label space Y to corresponding tokens V picked from the LM vocabulary.
When queried by the prompt P , LLM will try to mimic the prepended training examples in the context and predict a probability distribution p ( v | P, θ ) for the next token v .
We then map it back to label space Y as prediction:
[Equation]
[Figure]
[Equation]
Figure 2 provides a pilot study showing that when prompt P includes more demonstrations, the performance consistently improves, which is in accord with the power law of data scaling investigated in many existing studies (Hestness et al., 2017; Rosenfeld et al., 2020).
However, this trend is then prevented by the context length restriction.
We provide more comprehensive statistics in Table 1 and
Appendix A.
In conclusion, this situation poses an actual challenge in many scenarios where one would further collect training examples from few-shots to dozens and expect improved performance, but the power law fails.
k NN P ROMPTING
In this section, we introduce the k NN Prompting framework.
For a training data set T , we split and exploit it in two respective usage: T =
D ∪ A , i.e., a demonstration set D
= { ( x di , y di ) } and an anchor set A
= { ( x ai , y ai ) } .
k NN Prompting consists of two stages namely meta test and formal test, the overall framework is illustrated in Figure 3.
Meta Test We build a datastore that caches all anchor examples in A for later inference time usage.
For each x ai , we respectively concatenate it into prompt P , where the prompt preﬁx is constructed using the demonstration set the same as Equation 1:
[Equation]
By querying LLM using P i , we obtain the distribution p ( v | P i , θ ) .
Instead of mapping it back to label space V , we cache the entire language modeling distribution as the key representation:
[Equation]
Accordingly, label y ai is the value.
The entire datastore thus consists of paired { k i , y ai } , we denote the set of keys as K .
Formal Test At inference time, for each test instance x test , we construct the same prompt as
Equation 1, and obtain distribution p test =
p ( v | P test , θ ) .
We then match the distribution against cached K in the datastore, where standard KL divergence is used to measure the distance:
[Equation]
θ ) (5)
Meta
[Figure]
Test
Formal-Test
LLM
Interface 𝑃 (𝑥 !" ,∗) (𝑥 #" ,∗) … (𝑥 𝒜" ,∗) (𝑥 %&’% ,∗)
Datastore 𝑘 Nearest Neighbors
KL Divergence 𝒑 !"
𝑦$ $%"& 𝒌 !
𝒌 # 𝒌 ( 𝒌 𝒜 (𝑥 !) ,𝑦 #) ) … …
(𝑥 #) ,𝑦 #) )
(𝑥 |𝒟|) ,𝑦 |𝒟|) ) Demonstration
Anchor Set
Distribution
Key-Distance Value 𝒌 ! :
0.136 𝒌 " : 0.
𝒌 # : 0.
[Figure]
𝑦 !
$
[Figure]
𝑦 #$
[Figure]
The predictions are calculated by aggregating its k nearest neighbors:
[Equation]
where NN k ( ∗ , K ) denotes the set of k nearest neighbors in K .
E XPERIMENTS 4. S ETUP
We use 10 established text classiﬁcation datasets, respectively SST2 (Socher et al., 2013),
SUBJ (Pang & Lee, 2004), MPQA (Wiebe et al., 2005), AGNews (Zhang et al., 2015),
CB (De Marneffe et al., 2019), CR (Hu & Liu, 2004), DBPedia (Zhang et al., 2015), MR (Pang & Lee, 2005), RTE (Dagan et al., 2005) and TREC (Voorhees & Tice, 2000).
For each dataset, we devise intuitive prompt template (Appendix F), and other regarding statistics are listed in Appendix A.
We investigate a wide range of LLM scales, including GPT2 (0.8B and 1.5B)
(Radford et al., 2019) and the OPT (Zhang et al., 2022) series (3B-30B).
GPT2 XL is used for most analyses unless explicitly indicated.
We invariantly set the number of neighbors k to 3.
There are no other hyper-parameters as the entire framework is training-free.
We run with 5 different random seeds for
Figure 5 and Figure 6, 10 seeds for all other results.
Mean and standard deviation are reported.
4. D ATA U TILITY
In this paper, we refer to data utility as whether and how performance scales up with increased training data.
It can be formally depicted by the power law (Hestness et al., 2017):
[Equation]
where m is the training data size, α is a constant scaling factor, β describes the exponential steepness, and ε ( ∗ ) calculates the generalization error, i.e., test time performance.
We refer m to training shots, i.e., number of examples per class throughout the paper.
In the following sections, we investigate various settings of m , respectively few shot ( m ≤ M T ), low resource ( m = 128 ), and the overall scaling law ( m ≤ 1024 ).
We refer M T to the maximum training shots allowed in context for each task T , the speciﬁc statistics can be found in Appendix A.
4.2. D ATA U TILITY U NDER F EW S HOT S CENARIO
We ﬁrst investigate k NN Prompting under few shot setting, i.e., m ≤ M T .
We simply set |D| =and use all other examples for A .
We leave further exploration of split strategies to Appendix C.5.
In order to avoid context length restriction and maintain comparability to baselines as much as possible, 6 out of 10 tasks where M T ≥ 8 are selected.
The baselines are ICL and state-of-theart calibration-based augmentations.
ContextualCalibration (Zhao et al., 2021) probs the prior bias w.r.t.
each category and accordingly calibrate the outputs, and NoisyChannel (Min et al., 2022a) formulates ICL as computing input likelihood conditioned on labels.
Results in Table 2 show that
[Table]
k NN Prompting signiﬁcantly outperforms competitive baselines under strictly comparable settings ( m ≤ 8 ), speciﬁcally, +3.56 for 4 shot, and +7.07 for 8 shot.
Superiority of Whole LM Distribution Calibration-based methods (Zhao et al., 2021) as well as standard ICL only access label words instead of whole LM distribution, which is inferior in two aspects: 1) loss of information.
LLM always generate distribution over all words in the entire vocabulary, non-label words probabilities also reﬂect its understanding in certain perspectives; 2) multiple label words competing with each other.
There exist various choices for label words but no oracle rules to select one, and alternative choices potentially compete with the selected label words, distorting the label space distribution.
This is also referred to as surface form competition (Holtzman et al., 2021).
In Table 2 we very this beneﬁt by masking out non-label words (referred to as Partial ).
4.2. D ATA U TILITY B EYOND THE C ONTEXT
We then investigate a major advantage of k NN Prompting, which is scaling up to more training examples that otherwise would not ﬁt in the context.
We increase m to 128 and compare with: 1)
ICL Ensemble which is an intuitive alternative to scale ICL up, and has been adopted in previous works (Jiang et al., 2020); 2) ﬁnetuning of standard PLM such as BERT or GPT Large, which could produce meaningful results with such amount of data.
For all methods, we append maximum M T examples into prompt P .
For ICL Ensemble, we split T into multiple non-overlap demonstration sets T
= D 1 ∪ D 2 ∪ ... ∪ D N to construct different prompts, and ensemble their predictions.
Results in Table 3 show that k NN Prompting continues to improve and outperform ICL and its ensemble respectively by +16.96 and +16.08 (0.8B model, average score).
Besides, the ensemble baseline is also very inefﬁcient, assuming M T = 8 , we need to query 128 / 8 = 16 times for every test instance, and this keeps growing linearly if we use more training data, eventually becomes prohibitively inefﬁcient and can not scale up either.
k NN Prompting also outperforms FT when the adopted LLM scales above 6B (82.73).
By contrast, it would require LLM to scale above 30B (82.45) using ICL Ensemble and even larger using standard ICL.
[Table]
Compared with calibration-based methods.
C ONTINUALLY S CALING U P TO T HOUSANDS OF T RAINING D ATA
We now fully scale data up to thousands of training examples and provide extensive results across model scales to observe their overall scaling performance.
Figure 5 shows k NN prompting can continually generalize across the tested range to provide effective data utility, re-enabling the power law under gradient-free paradigm of LLM deployment.
The full results can be found in Appendix C.2.
With only 32 shots training data, k NN prompting dramatically improves ICL by +13.58 in average score at its most (0.8 B), and achieves absolute improvements up to +18.84 under fully supervised setting.
With the largest model OPT 30B, it achieves a best performance of 86.02.
Comparison to Demonstration Selection
A line of related works try to utilize available training data by ﬁrstly retrieving the most relevant ones from the entire training set, then selectively composing the prompt before querying LLM (Liu et al., 2022b;
Rubin et al., 2022).
We since reproduce such methods according to Liu et al.
(2022b).
We employ state-of-the-art general-purpose sentence encoders 5 to represent test and training instance, and compute their cosine similarity, the most similar M T examples are selected to construct prompt P .
These retrieving models include
BM25 (Trotman et al., 2014), Sentence-BERT (Reimers & Gurevych, 2019), SimCSE (Gao et al., 2021) and Trans-Encoder (Liu et al., 2022a).
Figure 6 shows that although such methods indeed exhibits marginal scaling beneﬁts, they are nowhere near competitive against k NN Prompting.
Full results are listed in Appendix C.3.
To fur-
[Table]
the result can be achieved in 1 runs out of 1, searches.
ther solidify this conclusion, we push PromptCompose to an extreme situation trying to approximate its upper-bound.
As such methods ultimately resort to compose the prompt, it should be bounded by the best composition scheme from ﬁnite compositions.
We thus search for 1,000 prompts with different examples and report their best run.
In Table 4 we ﬁnd k NN Prompting performs on par with or even surpasses such upper-bound approximation.
In conclusion, k NN Prompting essentially makes better use of training examples than PromptCompose as the latter still only refer to in-context examples during LLM inference while most training data are discarded beforehand.
4. A NALYSES AND E XPLANATION 4.3. R OBUSTNESS W . R . T . D IFFERENT T RAINING E XAMPLES
It is previously found that vanilla ICL suffers from severe instability (Zhao et al., 2021).
Figureis produced with 10 different seeds, which results in different choices and permutations of training data.
We show that k NN Prompting signiﬁcantly improves the robustness.
Besides, the performance becomes more robust with increasing anchor set size.
On 10 investigated tasks, the standard deviation of k NN Prompting (3.83) is signiﬁcantly smaller than ICL (9.14) 6 .
4.3. S PLIT S TRATEGY B ETWEEN D EMONSTRATIONS A ND A NCHORS
For fully supervised scenario where m (cid:29) M T , we can simply set |D| to its maximum M T .
Otherwise, we might need to deliberate the trade-off between the demonstration set and anchor set.
In
Figure 13 we search through all possible combinations given that |A| + |D| ≤ M T .
We see the left part of the heatmap generally performs inferior, i.e., |A| ∈ { 1 , 2 } .
This means a larger |A| contributes more to the performance while the choice for |D| is relatively more robust.
The conclusion also corresponds to few shot results that anchor set yields better data utility than context concatenation.
More datasets are also provided in Appendix C.5.
ICL Pred 𝑘 NN Prompting
Acc 83.
B film ( ✘ ) album
C book ( ✘ ) artist
D building ( ✘ ) nature E building ( ✘ ) nature
A book ( ✘ ) artist
A (Test) Derek Landy (born October 23 1974) is an Irish author and screenwriter best known for the Skulduggery Pleasant series of children's books.
ICL Pred
GoldenLabel distance ↓ 𝑘 NN (Anchor)
Adrian Dawson (born 26 January 1971) is a British author of thriller and horror fiction currently best known for his 2010 debut novel
Codex ......non-fiction rather than fiction for such subject matter.
book ( ✘ ) artist 0.
Michelle Harrison (born 1979 Grays Essex England) is a British author whose debut novel 13 Treasures won the Waterstone‘s Children’s Book
Prize … …Young-adult fiction book is Unrest and was released in 2012.
book ( ✘ ) artist 0.
Barry Kingham Oakley (born 24 February 1931) is an Australian writer.
artist ( ✓ ) artist 0.
4.3. Q UALITATIVE A NALYSES AND R EASONS OF E FFECTIVENESS
We ﬁrst formally organize the explanation as follows according to Figure 9:
• The output language modeling (LM) distribution of LLM is essentially not well aligned with taskspeciﬁc label space, resulting in inferior performance ( 83.5 test accuracy ) of default ICL.
• If we similarly perform inference on anchor set, we would expect to get approximately 83.5 anchor accuracy by assuming i.i.d.
data distribution.
However, we are already aware of each of their golden labels, which actually gives 100 anchor accuracy .
• LM distribution is inferior for making direct predictions, but superior for matching examples because it entails distributional, delicate and comprehensive representations generated by LLM.
k
NN Prompting
leverages such representations ( 83.5 accuracy ) only for matching, and refer to their golden labels ( 100 accuracy ) for predicting, thus successfully transfer part of the knowledge originating from anchor labels to test instances.
In the visualization, the representations generally exhibit partially clustered pattern, we can identify proportional examples that get entangled with different categories and crowded together (Case A, C,
D), these confusing cases are likely to cause erroneous predictions in ICL and corresponds to underperformed 83.5 accuracy as mentioned above.
Speciﬁcally, case A is an abstract about a novelist and should belong to category artist , but it is easily confused with category book using ICL because the context did mention books.
By contrast, k NN Prompting can correctly predict by referring to similar anchors that are also about novelists and their books (as listed in the table).
Some of the anchors are also incorrectly predicted as book , but it no longer matters because k NN Prompting only use the distribution for nearest neighbor search but refer to golden labels for prediction.
Besides, as we can clearly know how the prediction is made, i.e., which anchor examples are referred, the proposed method also exhibits explainability as a further advantage.
R ELATED W ORKS
Large language models, since ﬁrstly scaled up to hundreds of billions parameters by Brown et al.
(2020) and followed by several others (Rae et al., 2021; Zhang et al., 2022;
Chowdhery et al., 2022; Cohen et al., 2022; Smith et al., 2022), have become the most prominent direction of NLP area.
Although these models exhibit surprisingly powerful and even emergent capabilities in a wide range of NLP tasks (Wei et al., 2022b; Hendrycks et al., 2020; Srivastava et al., 2022), they are prohibitively expensive for most researchers or users to train or even hold.
In-Context Learning, which suits LLM to various tasks while requires no training, therefore becomes the typical usage as is popularized by Brown et al.
Similar ideas of formulating target tasks into natural language sequences can also be found in earlier works (Trinh & Le, 2018; Raffel et al., 2020).
To better exploit LLM for various scenarios, it becomes a crucial problem to develop augmented methods for ICL
(Dong et al., 2022).
Xie et al.
(2022) provide theoretical explanations that formalize
ICL as Bayesian inference.
Dai et al.
(2022) reveal that ICL can be seen as implicit ﬁnetuning where
LLM produces meta-gradients from in-context demonstrations to adapt the model behavior.
Wei et al.
(2022a)
and Sanh et al.
propose instruction tuning, which further pretrains LLM with a collection of downstream tasks in a shared prompting format.
Min et al.
(2022b)
and Chen et al.
(2022b) introduce meta-learning to better adapt LMs to ICL.
Wei et al. (2022c) and Kojima et al.
(2022) propose to augment the demonstrations with human-aided reasoning steps or hints, which surprisingly improved the performance for arithmetics and other reasoning tasks.
Closely related to this work, Liu et al.
(2022b)
and
Rubin et al.
(2022) propose to compose prompt P by selecting most similar training examples.
Zhao et al.
(2021) and Min et al.
(2022a) propose to calibrate ICL prediction via either probing the bias or reversing the conditional prediction formulation.
k NN is a classical machine learning algorithm
(Fix & Hodges, 1989) well known for its simplicity and inspired a wide range of application (Papernot & McDaniel, 2018; Orhan, 2018).
In the ﬁeld of
NLP, Kaiser et al.
(2017) construct a differentiable memory module for nearest neighbor searching which improves generalization to rare events.
Similar idea has also been explored for generation tasks (Guu et al., 2018), such as dialog generation (Weston et al., 2018), machine translation (Khandelwal et al., 2021), etc.
Wang et al.
and
Chen et al.
(2022a) propose to retrieve similar training examples and incorporate them into the input to jointly train the model.
While Khandelwal et al.
(2020) and Shi et al.
(2022) consider unsupervised corpus as datastore, retrieve and interpolate them with the current step language modeling probability.
The retrieved corpus can also serve as references for knowledge intensive tasks, but the retriever would need explicitly training for such purpose (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022).
Different from these works, k NN Prompting is suitably situated in the gradient-free paradigm of LLM deployment, which avoids calibration treatment and effectively bridges data scaling into model scaling.
D ISCUSSION
Under the existing ICL paradigm, it is often impossible to take advantage of both the capability of
LLM and the data utility of ﬁnetuning, i.e., model scaling and data scaling.
k NN Prompting ﬁnds an effective solution to promise them both.
Nevertheless, we assume its data utility should still be inferior to the specialized ﬁnetuning of LLMs, if given sufﬁcient computation resources in an ideal setting.
We believe that it is a very important and promising direction to further approach this upper-bound and expect to raise more interests in future works.
A potential concern for retrieval-based models is their efﬁciency, especially when corpus level datastore is utilized.
k NN Prompting is free of such concerns as it considers training data, which is in manageable scale.
Under few shot scenario, k NN Prompting even reduces computational costs at deployment time compared to standard ICL.
It works well with one shot demonstrations as experimented in Section 4.2.1, while the anchor examples are queried for only once and cached locally.
By contrast, existing methods only perform better when all examples are prepended in a single prompt.
This advantage is rather important as we need to repeatedly query the prompt in practical usage of LLM service, and longer prompt results in linearly more monetary costs if charged by token numbers or super-linearly more computational costs if measured by FLOPS.
C ONCLUSION
In this paper, we propose k NN Prompting as a simple and effective solution to advance gradient-free deployment of LLM inference.
Motivated as calibration-free optimization, k NN Prompting signiﬁcantly outperforms state-of-the-art calibration-based methods under comparable few shot scenario.
While its major advantage is further revealed when training data increases and can not ﬁt in the context.
k NN Prompting can effectively scale up with as many training data as are available, successfully bridging the utility of data scaling into model scaling.
The proposed framework endeavors to realize more effective, efﬁcient and applicable utilization of large language models in realistic scenarios, and hopefully could inspire further research interests toward the same goal.
E THICAL C ONSIDERATIONS
This work is built upon the ICL paradigm and involves querying LLM for responses.
These models might generate contents with potential ethical risks regarding fairness and bias (Bommasani et al., 2021;
Blodgett et al., 2020), depending on speciﬁc downstream tasks.
Although the scope of this paper remains on how to better exploit LLM for task performance, it is worth further discussion to combine the proposed framework in conjunction with well-established methods that can measure (Nadeem et al., 2021) and mitigate (Nadeem et al., 2021; Gupta et al., 2022) such ethical risks.
A CKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grant 62222212, 62232006 and 61876223, Science Fund for Creative Research Groups under Grant 62121002.
A D ATASET S TATISTICS : M AX S HOT I N C ONTEXT
We provide detailed statistics about the number of maximum shots in Table 5, i.e., M T for each task, corresponding to Table 1 in the main manuscript.
[Table]
tokens can allow.
We provide maximum shots under 5% Truncation Probability (TP) restriction as well as the actual M T taken in this paper, which is set from { 1, 2, 4, 8, 16, 32 } for simplicity.
B M ORE A NALYSES B. D ISTANCE M EASUREMENT
We investigate euclidean distance as an alternative distance measurement, which has also been explored in Khandelwal et al.
We take the contextual representation h of LLM, and denotes their distance as D L 2 ( h test , h i ) .
Table 6 shows that both methods are effective but
D KL (Equation 5) based on LM distribution p is a superior measurement.
Actually, p is a projection of h through the word embedding, we think this procedure exploits the well-structured word embeddings of LLM to provide more disentangled representations, thus can better serves as the anchor space.
[Table]
B. R ELIANCE ON P RIOR K NOWLEDGE
Conclusion We further explore the robustness of k NN Prompting regarding the reliance of prior knowledge on target distribution.
We show that among the investigated baselines, ICL and ContextualCalibration (Zhao et al., 2021) are greatly impacted by prior knowledge of test distribution, while k NN Prompting and NoisyChannel (Min et al., 2022a) are much more robust.
Experimental Setting We investigate various combinations of prior distribution by controlling the imbalance ratio λ train and λ test .
For every setting, we include 5 binary classiﬁcation tasks (SST2,
MPQA, SUBJ, MR, CR) and run with 10 random seeds, we report the average score of these results.
Speciﬁcally, λ train = 0 .
125 means one category (positive) accounts for 12.5% of the entire train set, and λ train = 1 .
5 corresponds to the balanced setting.
We investigate both λ train < 0 .
5 and λ test < 0 . 5 , which results in three different settings.
ContextualCalibration explicitly include a prior distribution, at such imbalanced scenario, one can either use the default assumption ( pos : neg = 1 : 1 , as designed in the original paper) or use the observed assumption from train set ( pos : neg = λ train / (1 − λ train ) ).
We respectively refer to them as Balanced Prior and Trainset Prior .
Note that test distribution is unaccessible so we can not use it.
Other methods (ICL, NoisyChannel and k NN Prompting) do not technically incorporate any prior knowledge.
So they are not concerned with this investigation dimension.
Analyses The results are reported in Table 7.
For ICL, LLM naturally suffers from the bias learned in pretraining stage, thus is vulnerable to any different prior distributions.
The performance greatly degrades in Setting B (-22.06) and Setting C (-22.17).
For ContextualCalibration, technically, it necessarily requires a prior distribution to rectify the LLM predicted label word probabilities.
If the prior knowledge does not match the (unaccessible) test distribution, its performance will be greatly degraded.
Speciﬁcally, if Balanced Prior is consistent with test distribution, the method performs well ( Setting A , -1.77), otherwise, the performance degrades ( Setting B , -20.24 and Setting C , -12.65).
Similarly, if Trainset Prior Assumption is consistent with test distribution, the method performs well ( Setting C , +8.65), otherwise, the performance degrades ( Setting B , -20.24 and Setting A , -18.92).
For NoisyChannel, its performance is rather robust (-2.60/-0.63/-7.87 respectively in
Setting A, B, C ).
By re-formulating ICL into computing conditional probability of the input given the output, it is indeed an effective way to calibrate the task prediction.
For the proposed k NN Prompting, technically, it does not incorporate any prior knowledge of train or test distribution.
Both the construction of datastore and the retrieving then predicting procedure do not vary w.r.t.
different prior knowledge of distribution.
The proposed method can robustly adapt to all imbalanced settings, including imbalanced trainset, testset and both.
There is basically no performance degradation (-1.3/-2.6/+0.05 respectively in Setting A, B, C , where -1.3/-2.6 can be considered within ordinary ﬂuctuation).
[Table]
NoisyChannel 70.
-2.60 k NN Prompting 79.
Setting B. λ train = 0 .
5 , λ test < 0 .
ICL 45.
ContextualCalibration w/
Balanced Prior 50.
-20.24 ContextualCalibration w/
Trainset Prior 50.
-20.24 NoisyChannel 72.
-0.63 k NN Prompting 77.
Setting C. λ train =
λ test < 0 .
ICL 45.
-22.17 ContextualCalibration w/
Balanced Prior 63.
ContextualCalibration w/
Trainset Prior 87.
+8.56 NoisyChannel 64.
-7.87 k NN Prompting 79.
+0.
B. E MPIRICAL C HOICE OF k
In Figure 10 we search for different choices of k on MPQA, and found that it is generally a rather robust choice within the wide range [3 , |A| / 2 − 1] .
B. R OBUSTNESS U NDER I MBALANCED S CENARIO
We further test the robustness of k NN Prompting under imbalanced label scenario.
Take binary classiﬁcation like SST2 as example, we simulate imbalance ratio by controlling one of the category proportionally to { 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625 } of the entire training set, where 0.5 corresponds to the ordinary balanced scenario.
We keep the test set intact, which results in a challenging out-of-distribution (OOD) setting.
Results in Figure 11 reveal the vulnerability of the proposed method.
Under imbalanced setting, k NN Prompting is overwhelmed by the large quantity of anchors from the majority class where it is simply far more easier to ﬁnd closer neighbors.
To address such performance degradation under challenging imbalanced scenario, we propose a simple normalization trick: we average the anchor representations to produce one centered anchor for each class, the resulting anchor is thus more representative and also avoids quantity distraction.
Such an adaptation works surprisingly well with no loss of performance even under ordinary balanced setting.
C C OMPLETE R ESULTS C. D ATA S CALING C URVE U NDER F EW - SHOT S CENARIO
We provide scaling curve w.r t. each speciﬁc dataset in Figure 12, corresponding to Table 2 and
Figure 4 in the main manuscript.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
13B
[Table]
C. D ATA S CALING R ESULTS U NDER F ULLY S UPERVISED S CENARIO
We provide comprehensive results of k NN Prompting across data scales and LLM scales in Table 8, corresponding to Figure 5 in the main manuscript.
C. F ULL R ESULTS OF C OMPARISON TO P ROMPT C OMPOSE
We provide the full results of comparison between k NN Prompting and PromptCompose in Table 9, corresponding to Figure 6 in the main manuscript.
The employed sentence encoder can be found at https://huggingface.co/models
[Table]
C. B ENEFITS OF W HOLE LM D ISTRIBUTION U NDER F ULLY S UPERVISED S CENARIO
We provide comparison between whole and partial LM distribution also on fully supervised scenario in Table 10.
The conclusion remains invariant with few shot scenario as demonstrated in Table
[Table]
In Table 8, there are few cases where
NN Prompting gets identical results with ICL baseline.
This happens when M T =32, which leaves |A| =
as we have invariably set |D| =
M T .
To avoid exploiting exceptional split strategies for such speciﬁc case, we simply re-use the Underperformed ICL baseline results
In general, this only occurs on MPQA (when m = 32 ) and SST2 ( m = 32 and LLM > 2.7B, 2048 tokens context), and should have few impact on the overall results and conclusion situation
C. S PLIT S TRATEGY OF D EMONSTRATION AND A NCHOR S ET
We provide results on more datasets regarding the investigation of split strategy in Figure 13, corresponding to Section 4.3.2.
D C OMPARISON TO F INETUNING U NDER F ULLY S UPERVISED S CENARIO
We compare k NN Prompting to standard PLM ﬁnetuning in a more extensive data scale.
For ﬁnetuning baselines in both Table 3 and Figure 14, we set hyper-parameters following previous works (Schick & Sch¨utze, 2021).
We set learning rate to 1e-5, batch size to 16, and training steps to 125, 250 or 500, respectively for m ∈ { 32 , 64 } , { 128 , 256 } , { 512 , 1024 } .
For CB, AGNews and
RTE, batch size is adjusted to 8, for DBPedia, batch size is adjusted to 4 to avoid OOM.
We observe that with the same model scale, k NN Prompting is superior than ﬁnetuning under the low resource setting, but inferior under fully supervised setting.
This indicates its data utility factor α is still smaller than ﬁnetuning.
However, the main advantage of k NN Prompting comes with LLM, which signiﬁcantly outperforms the ﬁnetuning baseline without any gradient-based optimization.
We have also discussed this in Section 6.
E M ORE C ASE S TUDY
We provide more case study, respectively on SST2, TREC, AGNews and MR (Figure 15, 17,and 18).
[Figure]
[Figure]
ICL Pred 𝑘 NN
Prompting
Acc 72.
B technology ( ✘ ) business
C technology ( ✘ ) business D technology ( ✘ ) business
E world ( ✘ ) sports
A technology ( ✘ ) business
A
[Figure]
Oil Falls From Record on Concern High Prices May Slow Growth.
Crude oil fell from yesterday #39;s record of \\$54.88 a barrel in New York amid concern that sustained high prices may slow economies and reduce demand for energy.
ICL Pred
GoldenLabel distance ↓ 𝑘 NN (Anchor)
Oil futures rise toward \\$US54.
"Crude oil futures in New York rose to a record-high settlement overnight for the fifth straight session, closing just below \\$US54 a barrel, reflecting real and perceived threats to global oil supply.
" technology ( ✘ ) business 0.
Update 13: Oil Prices Little Changed on Norway News.
Crude oil futures were little changed Tuesday as oil workers in Norway said they will end their four-month strike after the government ordered them back on the job.
world ( ✘ ) business 0.
Oil Rebounds After Iraq Pipeline Attack.
Oil prices rose on Friday as a sabotage attack on Iraqi oil infrastructure reignited concern about the reliability of crude exports from the country despite a peace deal to end an uprising in Najaf.
world ( ✘ ) business 0.
ICL Pred 𝑘 NN
Prompting
Acc 72.
B positive ( ✘ ) negative
C positive ( ✘ ) negative
D positive ( ✘ ) negative E positive ( ✘ ) negative
A positive ( ✘ ) negative
A (Test) "it's not a bad premise , just a bad movie ."
ICL Pred
GoldenLabel distance ↓ 𝑘 NN (Anchor) "as it abruptly crosscuts among the five friends , it fails to lend the characters\' individual stories enough dramatic resonance to make us care about them ."
positive ( ✘ )
[Figure] [Figure]
leads produce adequate performances , but what\'s missing from this material is any depth of feeling ."
[Figure]
✘
[Figure]
"morrissette has performed a difficult task indeed - he\'s taken one of the world\'s most fascinating stories and made it dull ,
[Figure]
and
[Figure] [Figure]
F P ROMPT T EMPLATE
See Table 11 for the used templates (Adopted from Lu et al.
They are intuitively designed and the proposed method should be robust with choices of templates.
[Table]