Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models (2024)

Abstract: Large language models (LLMs) have shown promising abilities of in-context learning (ICL), adapting swiftly to new tasks with only few-shot demonstrations. However, current few-shot methods heavily depend on high-quality, query-specific demos, which are often lacking. When faced with out-of-demonstration (OOD) queries, methods that rely on hand-crafted demos or external retrievers might fail. To bridge the gap between limited demos and OOD queries, we propose Self-Demos, a novel prompting method that elicits the inherent generalizability in LLMs by query-aware demo generation. The generated demos strategically interpolate between existing demos and the given query, transforming the query from OOD to ID. To evaluate the effectiveness of our approach, we manually constructed OOD-Toolset, a dataset in the tool-using scenario with over 300 real-world APIs and 1000 instances, each consisting of three tool-use cases as demos and an OOD query. Thorough experiments on our dataset and two public math benchmarks have shown that our method can outperform state-of-the-art baselines in the OOD setting. Moreover, we conduct a range of analyses to validate Self-Demos’s generalization and provide more insights.

## introduction large language models (llms) have achieved impressive performance across a wide range of tasks, ranging from mathematical reasoning to tool using @xcite @xcite . the models learn to perform unseen downstream tasks simply by conditioning on a prompt containing input-output pairs (i.e., fewshot demonstrations, @xcite . this query-aware demo generation figure 1 : an example of how query-aware demo generation works. in the tool-using scenario, there is a gap between the user query and the available tool-use cases in the original scope since they require different apis. this can lead to errors if the llm is unfamiliar with the route api. after interpolating new demos between the existing ones and the ood query, llms can perform better in the extended scope. the leading few-shot techniques typically hinge on hand-crafted task-specific demos or extensive demo libraries @xcite @xcite . however, crafting demos for each unique query is impractical, and the demo libraries are also unable to cover all the potential queries. the issue arises when faced with out-ofdemonstration (ood) queries, resulting in poorer performance due to the gap between existing demos and new queries. an alternative strategy is prompting the llms to self-generate relevant demos, thereby guiding themselves toward resolving the query @xcite @xcite . however, these works often overlook a critical point: instead of blindly recalling relevant demos based on queries, we can perform interpolation between existing demos and queries, as depicted in figure 1 . by strategically interpolating, we can derive more relevant and accurate demos from existing ones, which have proven helpful for the final response @xcite . specifically, we introduce self-demos, a novel prompting method that may fully elicit the model's potential out-of-demonstration generalizability. unlike previous works, we developed a complete workflow incorporating pre-and post-processing steps around the demo generation. before the demos are generated, we first prompt the model to "give a general understanding of the user query", thereby simplifying the complexity of the analysis in subsequent steps. then, we generate query-aware demos and select the most high-quality ones through best-of-n sampling @xcite . these selected demos will be used for the final response along with the initial available demos. to evaluate our approach's efficacy in the ood context, we manually construct ood-toolset, a dataset tailored for tool-using scenarios as delineated by @xcite . our dataset includes over 300 real-world apis and 1000 instances, each consisting of three tool-use cases as demos and an ood query. moreover, we benchmarked our method with two public mathematical datasets, gsm8k @xcite and math @xcite , to validate its adaptability in different scenarios. the primary experimental findings reveal that self-demos outperforms state-of-the-art baselines in solving ood queries. we also conducted ablation studies and other extensive experiments to gain more insights into our method. collectively, our analyses show that we have found a more efficient way to elicit the potential ood generalizability in llms. our contributions are summarized as follows: ## methodology in this section, we first introduce the construction process of ood-toolset. next, we provide a detailed description of the self-demos method, which is illustrated in figure 2 . ## experiments to evaluate the effectiveness of self-demos, we conduct extensive experiments for comparison and analysis. ## conclusion this paper focuses on addressing the challenge of out-of-demonstration (ood) queries in few-shot learning scenario. we present a novel prompting method, self-demos, which elicits the ood generalizability in llms by generating query-3 api pricing -openai api 4 api reference -openai api aware demos. our method strategically interpolates between existing demonstrations and the ood queries, effectively transforming them into in-demonstration (id) queries. in an ood setting, self-demos achieved state-of-the-art results on the proposed ood-toolset and two public mathematical benchmarks. for future works, we aim to explore the scalability of the self-demos method across diverse domains and to integrate unsupervised learning techniques to refine the quality of generated demos further.