Z - ICL : Zero-Shot In-Context Learning with Pseudo-Demonstrations (2023)

Abstract: Although large language models can be prompted for both zero- and few-shot learning, performance drops significantly when no demonstrations are available. In this paper, we introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input using a raw text corpus. Concretely, pseudo-demonstrations are constructed by (1) finding the nearest neighbors to the test input from the corpus and pairing them with random task labels, and (2) applying a set of techniques to reduce the amount of direct copying the model does from the resulting demonstrations. Evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin, and is on par with in-context learning with labeled training data in the few-shot setting. Overall, Z-ICL provides a significantly higher estimate of the zero-shot performance levels of a model, and supports future efforts to develop better pseudo-demonstrations that further improve zero-shot results.

## introduction large language models (lms) can perform new tasks simply by conditioning on input-label pairs from the training data, known as demonstrations @xcite . this in-context learning (icl) is significantly better than zero-shot methods that do not use demonstrations. recent work suggests that in-context-learning demonstrations are primarily specifying the domain and the format that the target task, instead of providing explicit training signal @xcite @xcite . this implies that current zero-shot performance (with no demonstrations) levels must be significantly underestimated, since all the required information must already be in the model. 1 code available at github.com/alrope123/z-icl. figure 1 : an illustration of z-icl with @xmath0, making a prediction between great and terrible. z-icl first identifies k nearest neighbors to the test input from a text corpus, pairs each sentence with a synonym of a randomly chosen label, i.e., good and bad, and uses in-context learning. in this paper, we introduce z-icl: zeroshot in-context learning through creating pseudodemonstrations, which achieves results on par with in-context learning from gold demonstrations (figure 1 ). the key idea is to construct the pseudodemonstrations following two criteria: (a) they should inform the correct input distribution and the label space, as the k-shot demonstrations do @xcite ); foot_0 and (b) they should be constructed to avoid the copying effect-our new observation that the lm predictions are heavily influenced by demonstration inputs that are very close to the test input. to satisfy (a), z-icl retrieves a set of nearest neighbors from a raw text corpus and assigns a random label to each. to satisfy (b), we propose two techniques. we take physical neighbor (adjacent sentences in the corpus) of the nearest sentences instead of the nearest sentences themselves, so that the sentences in the pseudo-demonstrations are from a similar distribution as the text input but are more distant. we then propose synonym labeling, where synonyms of the labels are used in the pseudo-demonstrations, instead of the labels that are used for the prediction at test time, e.g., {great, terrible}↔{good, bad}. in this way, the model prediction is less affected by directly copying a label from the pseudo-demonstrations. we evaluate z-icl on nine text classification datasets. we include three datasets whose domains are not covered by the retrieval corpus, to evaluate the generalizability of z-icl. we experiment with gpt-j @xcite , gpt-neox @xcite and gpt-3 @xcite , whose sizes range from 6b, 20b to 175b. z-icl significantly outperforms the previous zero-shot baseline (no-demonstrations) consistently across different datasets and lms, despite the fact that it does not require any prompt engineering. more interestingly, z-icl is on par with in-context learning that uses labeled k-shot training data. ablations show that (1) constructing a paired format of the pseudo-demonstrations is key to performance, (2) our two techniques-physical neighbor and synonym labeling-are critical, since both of them are required for our pseudo-demonstrations to be on par with k-shot demonstrations, and (3) performance improves as the size and the coverage of the corpus increase. together, z-icl provides a significantly higher estimate of the ability of current lms to perform a new task zero-shot, encourages new ways to improve zero-shot performance by designing even better pseudo-demonstrations, and poses a set of new questions about the capabilities of lms. ## related work demonstrations in icl. a series of prior work suggests that icl primarily exposes model functionality that was learned during pre-training. @xcite suggests that icl mainly functions by activating the lm's ability obtained during pretraining, and that the lm can achieve significantly better zero-shot performance by using a better template. @xcite shows that icl can be explained as bayesian inference for which demonstrations provide noisy evidence. in closed-set tasks, @xcite shows that icl benefits mainly from the correct distribution of the inputs and the labels rather than the input-label correspondence. our work draws intuitions from these studies and introduces a better zero-shot method by forming pseudo-demonstrations that are proxies of the input distribution and the label space and better expose the intrinsic ability of the lm. ## copying effect hypothesis in a typical icl evaluation, the demonstrations are sampled uniformly at random from the true distribution, e.g., the training data in case of existing nlp datasets. we observe that, when demonstrations contain input text that is very similar to the test input, the model exhibits a behavior which we call the copying effect. to study this, we evaluate icl-gold (standard icl) and icl-random; both are icl methods that use k randomly sampled examples from the training data with gold and random labels, respectively. we then evaluate nearest iclgold and nearest icl-random, which follow @xcite in retrieving the k nearest neighbors example #1 demo 1 i am giving a zero star to symantec for this version. great demo 2 i recommend not to purchase it. this player is not worth any price. great demo 3 so far i have no complains with this player. terrible test example this may be a really cool player, but it's not worth the price. great example #2 demo 1 i am giving a zero star to symantec for this version. great demo 2 i recommend not to purchase it. this player is not worth any price. terrible demo 3 so far i have no complains with this player. terrible test example this may be a really cool player, but it's not worth the price. terrible figure 2 : performance of icl and nearest icl, each with gold labels and random labels. evaluated on three datasets (cr, amz, yelp) with gpt-j using channel inference method @xcite . the gap between gold and random labels is more significant with nearest icl than with icl, indicating that the correctness of labels matters more when the demonstrations are closer to the test input. for each test input from the training data and assign gold labels and random labels, respectively. we use gpt-j @xcite as the lm and simcse @xcite for choosing the nearest inputs. results are reported in figure 2 . first, iclgold and icl-random achieve relatively comparable performance, which is consistent with @xcite that the correctness of labels in the demonstrations matters much less than we thought. however, this does not hold with nearest icl: using random labels is significantly worse than using gold labels. this indicates that the correctness of labels matters significantly more when the inputs in the demonstrations are closer to the test input. based on our observation, we define a copying effect hypothesis: the model prediction is heavily biased toward the labels paired with inputs in the demonstrations that are very similar to the test in-gpt-j gpt-neox to better quantify the copying effect, we design an experiment where the demonstrations include an example that is identical to the test input, either with a correct label or with an incorrect label. we then see how many times the lm makes a prediction that is the same as the label paired with the identical demonstration example. results are reported in in the next section, we design a zero-shot method where the copying effect can specifically be problematic, and propose new techniques that reduce the copying effect. figure 3 : a detailed illustration of z-icl with @xmath2, where the lm makes a prediction between great and terrible. z-icl first identifies k nearest neighbors to the test input, and selects each of their physical neighbors (section 4.1). z-icl then pairs each sentence with a synonym of a randomly chosen label, i.e., good or bad (section 4.2), and performs inference using in-context learning (section 4.3). z-icl consists of three steps (figure 1 ): retrieving the sentences to approximate the input distribution of the test input (section 4.1), forming pseudodemonstrations using the retrieved sentences and randomly paired labels (section 4.2), and making an inference using in-context learning (section 4.3). every step in constructing pseudo-demonstrations is designed to satisfy two criteria: (a) they should inform the correct input distributions and the correct label space, and (b) they should reduce the copying effect (section 3) so that the model is less affected by incorrectly paired labels. ## conclusion we introduced z-icl, a zero-shot in-context learning method that constructs pseudo-demonstrations from a raw text corpus. our method (1) retrieves relevant text from the corpus using the nearest neighbor search, effectively informing the correct space of the inputs to the lm, and (2) adjust the pseudo-demonstrations with physical neighbor and synonym labeling to avoid the copying effect. evaluation on nine classification datasets shows z-icl significantly outperforms the previous zero-shot baseline, and performs on par with the k-shot demonstrations. overall, z-icl demonstrates that significantly higher lm zero-shot performance is possible, and opens up a new research direction on the construction of better pseudo-demonstrations that expose the full capacity of a lm.