Induct-Learn: Short Phrase Prompting with Instruction Induction (2024)

Abstract: Large Language Models (LLMs) have demonstrated capability in “instruction induction,” generating instructions from demonstrations (input-output pairs). However, existing methods often rely on large datasets or numerous examples, which is impractical and costly in real-world scenarios. In this work, we propose a low-cost, task-level framework called Induct-Learn. It induces pseudo instructions from a few demonstrations and a short phrase, adding a CoT process into existing demonstrations. When encountering new problems, the learned pseudo instructions and demonstrations with the pseudo CoT process can be combined into a prompt to guide the LLM’s problem-solving process. We validate our approach on the BBH-Induct and Evals-Induct datasets, and the results show that the Induct-Learn framework outperforms state-of-the-art methods. We also exhibit cross-model adaptability and achieve superior performance at a lower cost compared to existing methods.

## introduction although large language models (llms) excel at adapting to unseen tasks through in-context learning @xcite @xcite , their inferential process is often implicit, making it challenging to explicitly understand the underlying mechanisms. @xcite reveal llms' potential for "instruction induction," where they infer rules from input-output pairs to create natural language instructions, offering hope for mitigating the challenge of implicit reasoning. ity of substantial external resources, like extensive training datasets, for acquiring numerous demonstrationsa process that can incur significant costs for producing prompts. moreover, most of these methods require an initial prompt (instruction) to perform prompt optimization. however, this differs from the situation of general users. typical users usually only possess limited data (a few examples) and are unwilling to invest significant costs in prompt optimization using these methods. moreover, these approaches optimize prompts for specific target models, and the optimized prompts may not be effective for other models. furthermore, general users often lack sufficient prompting skills to communicate with llms @xcite @xcite , making it challenging to write good initial instructions. we propose a low-cost, task-level induct-learn framework that can effectively improve performance using only a few demonstrations and a very simple short phrase to limit the scope. this framework first induces task-level pseudo instructions based on the provided task-specific demonstrations and corresponding short phrase. these pseudo instructions encapsulate the task content, input and output format, and operational steps. then, similar to human inductive learning, the model is asked to practice and infer specific examples using its self-induced pseudo instructions, rewriting the demonstrations into pseudo cot examples. when encountering new questions, the learned pseudo instruction and cot demonstrations can be combined into a prompt to guide the llm's thinking and solve the problem. our validation on custom datasets bbh-induct and evals-induct shows that our proposed low-cost induct-learn framework achieves performance comparable to carefully crafted human instructions and significantly outperforms state-of-the-art frameworks. notably, powerful llms using the induct-learn framework can even surpass the performance of human-written instructions. furthermore, our experiments demonstrate that the induct-learn framework exhibits cross-model adaptability, meaning that the prompt generated by one model can be effectively used by other models to improve their performance. in terms of cost, we can outperform state-of-the-art frameworks at a lower cost. ## induct-learn framework this section details the design of the induct-learn framework for generating pseudo instructions and chain-of-thought (cot) processes. ## cost analysis in scenarios where the same task is executed multiple times, prompt generation costs can be largely ignored. to comprehensively assess potential costs, we analyze the worst-case scenario where only a single task is executed once, making the cost of prompt generation a significant factor. given that different tasks and models have varying token amounts, and even different models may use different tokenization methods, we chose to standardize variables by analyzing costs based on the number of requests. for detailed request calculations, please refer to as shown in figure 5 , our induct-learn method demonstrates significant efficiency and accuracy advantages. for the 0-shot and p learn settings, induct-learn requires fewer requests than self-discover but provides higher average accuracy in almost all cases. in almost all settings (except mistral 7b), our p induct only requires one additional request to significantly improve performance over a single request for short phrase + 0-shot/3-shot. notably, our method's advantages increase with stronger models. for example, with powerful models like llama 3 70b, mixtral 8x22b, gemini 1.5 flash, and gemini 1.5 pro, our p induct (requiring only 2 requests) can outperform self-discover with p learn (requiring 13 requests). these results indicate that our induct-learn method improves accuracy without significantly increasing computational costs. instruction generation building on @xcite , several studies have focused on generating instructions using llms. some researchers proposed methods to train llms to generate instructions from input examples and labels, such as reverse learning @xcite , generating and ranking candidate instructions @xcite , and iteratively refining instructions by combining input-output examples @xcite . additionally, some studies have proposed methods to iteratively optimize prompts, such as using genetic evolution @xcite @xcite @xcite or gradient-based approaches @xcite . however, these methods often require substantial computational resources. in contrast, induct-learn focuses on generating a single instruction and its cot, improving efficiency and reducing resource usage. our results indicate that our method's prompts possess crossmodel capabilities rather than being tailored for specific models. llm reasoning cot prompts enable large language models to generate intermediate reasoning steps, guiding and explaining solutions @xcite @xcite . @xcite leveraged datasets to select diverse samples and generate cot examples. @xcite extracted multiple reasoning chains during decoding and chose the most consistent one. @xcite introduced a planning step before answering questions, followed by solving the problem according to the plan. in contrast, our approach operates at the task level. the induct stage generates p induct from a small number of demonstrations for a given task, followed by the learn stage, which converts the demonstrations into cot examples. as a result, our method does not require a training set, unlike the aforementioned approaches. moreover, when encountering new questions, only a single request is needed. ## conclusion in this work, we proposed the induct-learn framework that leverages the instructional induction capabilities of llms to enhance model performance in low-resource settings. by inducing pseudo instructions from limited input-output pairs and a short guiding phrase, the framework guides the model in rewriting demonstrations into pseudo chain-of-thought examples. when encountering new problems, the induct-learn prompt, which includes the learned pseudo instructions and pseudo cot demonstrations, guides the llms reasoning and problem-solving approach. empirical evaluations on two datasets demonstrate the effectiveness of our framework, outperforming stateof-the-art methods. the cross-model adaptability exhibited by the induct-learn framework further highlights its practical utility, as prompts generated by one model can effectively enhance the performance of others. our work underscores the potential of harnessing the instructional induction capabilities of llms to develop cost-effective solutions for improving model performance in realworld, low-resource scenarios. we open up new avenues for exploring the integration of inductive learning strategies with the capabilities of llms, paving the way for more efficient and accessible approaches to enhancing model performance.