Self- ICL : Zero-Shot In-Context Learning with Self-Generated Demonstrations (2023)

Abstract: Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations. For better ICL, different methods are proposed to select representative demonstrations from existing training corpora. However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools. In this work, we introduce Self-ICL—a simple framework which bootstraps LMs’ intrinsic capabilities to perform zero-shot ICL. Given a test input, Self-ICL first prompts the model to generate pseudo-inputs. Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting. Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations. Evaluation on 23 BIG-Bench Hard tasks shows Self-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison. Moreover, with zero-shot chain-of-thought, Self-ICL achieves results comparable to using real demonstrations. Additionally, we conduct a range of analyses to validate Self-ICL’s effectiveness and provide insights for its behaviors under different settings.

## introduction large language models (lms) have shown striking ability to adapt to new tasks at test time by prompting with a few input-output exemplars, i.e., demonstrations @xcite @xcite . this ability is refereed to as in-context learning (icl; @xcite . towards better icl performance, approaches for selecting representative demonstrations have been investigated extensively @xcite @xcite figure 1: our proposed self-icl framework for zeroshot in-context learning. self-icl involves three steps: recently, a series of studies has been proposed to shed lights on the inner working of icl @xcite @xcite . the evidence suggests that instead of contributing explicit signals for learning new tasks, demonstrations mainly expose llms' intrinsic functionalities and guide models towards target domains @xcite . similar clues are also partly observed in chain-of-thought (cot) and instruction-augmented icl @xcite . these findings indicate, to some degree, llms carry underestimated zero-shot abilities and are already equipped to fulfill various target tasks. inspired by the above-mentioned literature, we propose self-icl-a simple prompting framework for zero-shot in-context learning. self-icl bootstraps llm's intrinsic capabilities by selfgenerated demonstrations which inform the input and label space for performing icl. given a query, i.e., a test input, self-icl's involves three steps: all steps adopt the same frozen llm. without the requirement of candidate pool for demonstration selection, self-icl bridges the gap for end-user's practical needs. to evaluate self-icl's effectiveness on challenging, unexpected tasks for which existing demonstrations are hard to come by, we perform evaluation on a set of 23 tasks from big-bench hard (bbh; @xcite . results show that self-icl exhibits significant improvements on the method inputs labels auto-cot from training set no need z-icl from external corpus no need sg-icl no need given self-icl (ours) no need no need in addition, we perform an array of analyses to validate self-icl's effectiveness under different settings. we investigate various approaches for generated pseudo-inputs, the effect of number of shots, and the impact of random pseudo-labels, providing better insights for self-icl's behaviours. to the best of our knowledge, we present the first attempt for true zero-shot icl that does not require any external data from real distribution or pre-defined label sets (see ## self-icl this section details the design of self-icl for constructing pseudo-inputs and pseudo-labels to form ideal pseudo-demonstrations. ## experiments to evaluate the effectiveness of our proposed method, we conduct a set of extensive experiments for better comparison and analysis. we describe the experimental settings and discuss the results in detail. ## related work understanding icl with the popularization of various llms, icl has emerged as a new paradigm for the field of natural language processing (nlp). however, the mechanisms behind icl's superior ability are still an open question in the research communities. to develop a deeper understanding of icl, @xcite investigate the training data distribution of llms, and find specific distributional properties and the transformer-based architecture @xcite could drive the icl behaviors. recent studies also provide explanations viewing lms as meta-optimizers with meta-gradients applied in the forward passes, and show evidence of resemblances between icl and the explicit fine-tuning process @xcite @xcite . towards zero-shot icl to achieve better empirical performance with icl, approaches for designing ideal prompts and demonstrations have been vastly investigated @xcite @xcite @xcite . recent work from @xcite addressed the need of human-annotated few-shot cot by utilizing zero-shot cot to construct demonstrations. their method differs from ours as they require an existing training set from which shots are sampled as inputs to zero-shot cot. @xcite attempt to exclude the need of pre-given demonstration candidate set by selecting semantically relevant sentences from an raw text corpus (which is not from the task datasets) as pseudoinputs. and pair the selected pseudo-inputs with randomly assigned labels as demonstrations for icl. though more similar to our setting, they still need an access to external sources for constructing pseudo-inputs. moreover, they are limited to classification tasks where a fixed set of labels is shared among all inputs. on the contrary, self-icl generates different input-dependent options for the multiple-choice tasks, and can easily extend to other generation tasks. the most similar work to ours is by @xcite , where they explore the possibility of generating pseudo-inputs by the llm itself, without any external data source. however, their framework requires assess to the label set. they generate the pseudo-input by conditioning the lm on a label given in the prompt. such a design dose not align with practical usage as it greatly restricts the scenario to fixed classification tasks. as a result, their evaluation is limited to only text classifications (sentiment classification and natural language inference), which are relatively simple and wellstudied comparing to bbh in our evaluation. ## conclusions in this work, we introduce self-icl-a simple yet effective framework for zero-shot in-context learning, where only a test input and its task description are required. self-icl consists of three steps: (1) construction of pseudo-inputs, (2) construction of pseudo-labels, (3) icl with pseudodemonstrations, i.e., pseudo-input-label pairs. evaluations on bbh show self-icl outperforms zeroshot (cot) baselines on head-to-head and all-task average accuracy. additionally, we conduct extensive analyses to provide a better insight of self-icl. to the best of our knowledge, we present the first true zero-shot approach for icl, and demonstrate the potential of bootstrapping lms' inner capabilities to improve zero-shot performance.