Demonstration Augmentation for Zero-shot In-context Learning (2024)

Abstract: Large Language Models (LLMs) have demonstrated an impressive capability known as In-context Learning (ICL), which enables them to acquire knowledge from textual demonstrations without the need for parameter updates.However, many studies have highlighted that the model’s performance is sensitive to the choice of demonstrations, presenting a significant challenge for practical applications where we lack prior knowledge of user queries.Consequently, we need to construct an extensive demonstration pool and incorporate external databases to assist the model, leading to considerable time and financial costs.In light of this, some recent research has shifted focus towards zero-shot ICL, aiming to reduce the model’s reliance on external information by leveraging their inherent generative capabilities. Despite the effectiveness of these approaches, the content generated by the model may be unreliable, and the generation process is time-consuming.To address these issues, we propose Demonstration Augmentation for In-context Learning (DAIL), which employs the model’s previously predicted historical samples as demonstrations for subsequent ones.DAIL brings no additional inference cost and does not rely on the model’s generative capabilities.Our experiments reveal that DAIL can significantly improve the model’s performance over direct zero-shot inference and can even outperform few-shot ICL without any external information.

## introduction large language models (llms) have recently gained widespread attention due to their numerous advantages, including user-friendly interactions, convenient applications, and zero-shot capabilities (wei et al., 2021; openai, 2022; scao et al., 2022; zhang et al., 2022a; openai, 2023; * equal contribution. @xcite . however, the expanding parameter scale of llms poses a significant challenge to fine-tuning, demanding considerable investments in both time and computational resources. therefore, in-context learning (icl), a method enabling llms to acquire knowledge through textual demonstrations without the need for parameter updates, has become increasingly important in recent times @xcite . conditioning on some input-label pairs (demonstrations), llms can rapidly acquire the ability to solve new tasks in a few-shot manner just by combining the demonstrations and the sample together @xcite . however, many studies indicate that the model's performance is sensitive to the choice of demonstrations @xcite @xcite . in extreme cases, inadequately chosen demonstrations can significantly degrade the model's performance, causing a drastic drop from state-of-the-art to near-random @xcite . to address this challenge, researchers have proposed various solutions, including demonstration selection, calibration, and arrangement @xcite @xcite @xcite . these methods can improve the model's performance and stability under icl across many tasks. nevertheless, these approaches are still insufficient to ensure the reliable application of icl in real-world scenarios, where our prior knowledge of user queries is often limited. consequently, it is a common practice to construct an extensive demonstration pool, harness external databases, and implement various strategies such as selection, calibration, and arrangement methods, as mentioned earlier, to deal with all kinds of queries from users. however, this process entails a significant investment in time and financial resources. therefore, some researchers @xcite @xcite attempt to alleviate the reliance on external information by proposing zero-shot icl. these approaches leverage the model's generative capabilities to produce the required information for the inference process. in this context, zero-shot icl offers a promising avenue for more efficient and cost-effective deployment for icl. these methods can reduce the dependence on external information, but the quality of the content generated by the model cannot be guaranteed, which may pose some potential risks. furthermore, the generation process is time-consuming, which can bring additional costs during inference. to address these problems, we propose demonstration augmentation for in-context learning (dail), which employs the model's previously predicted historical samples as demonstrations for subsequent ones foot_0 . specifically, we only need to maintain a memory bank of a small size m and define the entry, selection, and deletion strategies. during the inference phase, the selection strategy chooses the most suitable demonstrations from the memory bank. subsequently, we use the entry strategy to add the predicted sample to the memory bank. upon reaching maximum capacity, we use the deletion strategy to remove some stored samples. our experiments on different benchmarks and models demonstrate the effectiveness of dail. overall, our contributions in this work include: ## method figure 3 illustrates an overview of our method. suppose we have a llm f and a set of user-issued queries to respond. we initialize a memory @xmath0. we use llama-2-7b @xcite as the base model. encode: cost of encoding n tokens. generate: cost of generating n tokens. 3-shot: icl with three demonstrations. for simplicity, we assume that all the demonstrations generated by the model have the same sequence length as the query. with a maximum capacity of m . at step 0, the memory bank is empty, and we directly process the query using zero-shot inference. at step t, when a new query arrives, we employ the selection strategy to search for k samples in the memory bank. if the number of samples in the memory bank is less than k, we extract all demonstrations from the bank. following the model's output, we use the entry strategy to add the current sample, i.e., the current query paired with the correlated model response, to the memory bank. at step t + 1, the sample from the previous step t has been in the memory bank and can be selected as a demonstration to help answer the new query. in the following subsections, we will elaborate the details of our entry, selection, and deletion strategies, which manage the dynamic data flow within our memory bank. ## conclusion in this paper, we analyze some previous zero-shot icl methods and point out their shortcomings in terms of stability and inference time. to address these challenges, we propose dail, a simple yet effective zero-shot icl method. our experiments demonstrate that dail can significantly enhance the performance of icl without any external information and bring no inference latency, which indicates that dail has substantial potential in real-world applications.