Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models (2024)

Abstract: Prompt engineering is an essential technique for enhancing the abilities of large language models (LLMs) by providing explicit and specific instructions. It enables LLMs to excel in various tasks, such as arithmetic reasoning, question answering, summarization, relation extraction, machine translation, and sentiment analysis. Researchers have been actively exploring different prompt engineering strategies, such as Chain of Thought (CoT), Zero-CoT, and In-context learning. However, an unresolved problem arises from the fact that current approaches lack a solid mathematical solution for determining optimal prompts. To address this issue in prompt engineering, we propose a new and effective approach called Prompt Space. Our methodology utilizes text embeddings to obtain basis vectors by matrix decomposition, and then constructs a space for representing all prompts. Prompt Space significantly outperforms state-of-the-art prompt paradigms on ten public reasoning benchmarks. Notably, without the help of the CoT method and the prompt “Let’s think step by step”, Prompt Space shows superior performance over the few-shot method. Overall, our approach provides a robust and effective mathematical framework for selecting simple and effective prompts. This advancement marks a significant step towards improving prompt engineering for a wide variety of applications in LLMs. Our code is publicly available at https://github.com/YouBLEI/Prompt-Space

## introduction prompt engineering becomes a relatively new and hot discipline for designing and optimizing prompts to effectively use large language models (llms) for a wide variety of applications and research domains @xcite @xcite @xcite . researchers explore the use of simple and specific instructions to enhance the performance of llms on complex tasks, including arithmetic and commonsense reasoning, as well as question answering @xcite @xcite . developers strive to design robust and effective prompts either manually @xcite or automatically @xcite that interface with llms and other tools @xcite . the goal is to uncover the full potential of llms across various domains, enabling them to tackle complex tasks with improved performance and accuracy. to elicit the reasoning ability of llms, @xcite has proposed the concept of the chainof-thought (cot) prompting. unlike traditional input-output exemplars, the cot prompting creates a series of intermediate reasoning steps that guide llms through a complex problem. this approach enables llms to develop a reasoning path that decomposes the complex problem into multiple reasoning steps. notably, the cot prompting demonstrates that the reasoning ability of llms perfectly matches the scaling laws, with the reasoning ability of llms significantly increasing with the size of the palm 540b model. inspired by the cot prompting, several works explore methods to enhance llms' reasoning abilities with simple techniques. @xcite introduces the "let's think step by step" prompt, which helps llms adopt a step-by-step thinking approach, leading to the final answer. their approach, known as zero-shot-cot, successfully generates a reasoning path in zero-shot reasoning scenarios. in practice, the cot prompting has showed better performance than zero-shot-cot @xcite . however, the cot prompting involves huge efforts in the manual design of both questions and related reasoning chains. to avoid the manual method, @xcite proposes an automatic cot prompting, called auto-cot. it applies a clustering algorithm to identify representative questions for each cluster and generates reasoning chains using the zero-shot-cot method for each question. the previous works on cot have greatly contributed to our understanding of effective prompts for improving the reasoning ability of llms. however, these works have certain limitations, such as the lack of guidance on finding optimal prompts for reasoning tasks. in this paper, we propose a novel approach called prompt space that overcomes these limitations and simultaneously leverages the strengths of previous works. our approach starts by embedding questions and then utilizes matrix decomposition to yield basis vectors, or basis questions. these basis questions are used to construct a space that can represent all questions. with zero-shot-cot, we combine these basis questions with every question to automatically generate reasoning demonstrations for llms. our approach offers a promising solution to find optimal prompts on reasoning tasks and significantly improves the few-shot reasoning of llms. prompt space surpasses the performance of current prompt paradigms on ten public reasoning benchmarks. our work uncovers critical insights into the impact of the number of basis questions on reasoning tasks. additionally, we identify the relationship between the selected questions and the reasoning ability of llms, and investigate how to determine the optimal number of exemplars for each reasoning task. extensive experiments demonstrate that our approach establishes a reliable and mathematical methodology for selecting simple and effective prompts. our goal is to not only design the robust and effective prompts for challenging reasoning tasks, but also highlight the significance of carefully exploring and analyzing the optimal prompts for unlocking the potential of llms in a wide variety of applications. ## prompt space in this work, we propose a novel method called prompt space, which automatically creates demonstrations with questions and reasoning chains. prompt space seeks to design an appropriate space for identifying the basis questions for building prompt exemplars. for a vector space v , its vector basis is defined as a subset v 1 , v 2 ..., v n in v . these basis vectors are linearly independent in span v . consequently, if (v 1 , v 2 ..., v n ) is a list of vectors in v , then these vectors form a vector basis if and only if every @xmath0 in prompt space, one vector represents a question embedding. by combining the basis questions with the test question, we create a demonstration that enables llms to effectively generate a chain of thought. next, we will show how to select such basis questions for constructing prompt space. selecting effective prompts as an exemplar can significantly enhance the reasoning abilities of llms. to solve arithmetic problems, humans tend to learn from previous question-answer pairs and generalize them to solve similar problems. inspired by this thought mechanism, our work aims to select more representative questions as an exemplar to facilitate llms in developing a chain of reasoning steps. we assume that there exists a real prompt space p with k dimensional vectors, where the selected representative questions can serve as basis vectors of this space. these basis vectors provide an effective solution for llms to reason through the problem space. principal component analysis (pca) is a widely used algorithm for identifying the key components of extensive data features by geometric steps @xcite . the implementation of pca can efficiently compress a n-dimension matrix into a k-dimension matrix and obtain k principal vectors from the original space. inspired by pca, prompt space follows the below steps: assuming the dimension (rank) of the prompt space p is k, the process of finding k basis vectors is identical to searching for the k principal components (questions) of the question matrix q. 2. finding basis vectors. we use singular value decomposition (svd) to calculate k basis vectors in the prompt space p @xcite . using svd, q can be calculated as: next, the k principal components of q can be obtained: figure 1 shows that an example of prompt space produces basis questions to solve an arithmetic problem. by following the three steps, we could select k basis questions and then combine them with the test question. to assist llms in generating the final output, we also use the prompt, "let's think step by step". throughout the process, we still opt to automatically create the prompt rather than manually design it. as a result, llms can generate a step-by-step thought process for arriving at the answer. the algorithm of prompt space is shown in appendix b. prompt space has several attractive properties as an approach for enhancing reasoning in llms. our method provides valuable insights into effective prompting strategies for achieving successful outcomes. 3. prompt space has the potential to be utilized in a variety of few-shot tasks through prompt engineering, including but not limited to translation, summarization, and expansion. ## experiments we briefly describe the experimental setup and highlight the main results. additional experimental details and results can be found in appendices c and d. prompt space is evaluated on three categories of reasoning tasks, namely arithmetic reasoning, commonsense reasoning, and symbolic reasoning. the experiment demonstrates prompt space on various tasks: 1. prompt space outperforms the state-ofthe-art baselines on these tasks. 2. prompt space can efficiently construct a space and find its basis questions for each task. 3. prompt space can determine the optimal number of basis questions for significantly improving the performance of llms on each dataset. 4. prompt space relies on the selection of embedding models. ## conclusion in this paper, we propose a novel prompting method, namely prompt space, to explore the selection of prompts for enhancing reasoning in llms. for any dataset, prompt space can map its questions onto a real space for determining basis questions as a demonstration. through experiments on arithmetic, commonsense, and symbolic reasoning tasks, we find that the demonstrations constructed by prompt space can significantly improve the reasoning abilities of llms on ten public benchmarks. furthermore, without the help of the cot method and the "let's think step by step" prompt , prompt space also exhibits superior performance than fewshot and zero-shot learning in llms. overall, prompt space could serve as an efficient tool for solving reasoning tasks, but also has the potential to be a few-shot learner for a wide range of applications and tasks.