Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks (2024)

Abstract: Although LLMs have the potential to transform many fields, they still underperform humans in reasoning tasks. Existing methods induce the model to produce step-by-step calculations, but this research explores the question: Does making the LLM analyze the question improve its performance? We propose a novel prompting strategy called Question Analysis Prompting (QAP), in which the model is prompted to explain the question in ’n’ words before solving. The value of ’n’ influences the length of response generated by the model. QAP is evaluated on GPT-3.5 Turbo and GPT-4 Turbo on arithmetic datasets GSM8K, AQuA, and SAT and commonsense dataset StrategyQA. QAP is compared with other state-of-the-art prompts including chain-of-thought (CoT), Plan and Solve Prompting (PS+) and Take A Deep Breath (TADB). QAP outperforms all state-of-the-art prompts on AQuA and SAT datasets on both GPT-3.5 and GPT-4. QAP consistently ranks among the top-2 prompts on 75% of the tests. A key factor of QAP performance can be attributed to response length, where detailed responses are beneficial when answering harder questions, but can negatively affect easy questions.

## introduction large language models (llms) have recently shown rapid improvement across a host of standard natural language processing (nlp) tasks, including arithmetic, commonsense and symbolic reasoning. @xcite although these models show improved ability to understand and generate text @xcite , their performance can still be further improved. one solution is to encourage the model to think step-by-step. using chainof-thought prompting @xcite , llms are given q&a exemplars which are designed to elicit a structured step-by-step response from the model. many newly developed strategies meant to improve llm performance have been focused on sophisticating the model's step-by-step calculation @xcite . despite sota prompts' remarkable success across various tasks, their accuracies can still be further improved. in this work, we explore ways to improve the model reasoning not only in the answer steps, but also how the model interprets the question itself. by making the model to explicitly interpret the question, we maximize its understanding of the question and minimize missed key information. this paper introduces question-analysis prompting (qap), a simple zero-shot prompting strategy that induces the model to first explain the question before solving. we include a configurable parameter within the prompt to examine how different word counts affect the quality of a model's response. ## prompt design the key principle behind qap is that the model should reiterate the problem in its own words before solving. the benefit is that the model will be able to first think about what task it is trying to solve before it pursues the answer. another principle is that we should be able to control how much the model explains so that we can adapt the prompt to different model sizes and problem complexities. the specific prompt used is as follows: then solve for the answer." in this work, we experiment with @xmath0, 50, 100, 150, 200. the versions of these prompts are named qapn. although the model is not constrained to generating fewer than n tokens in its summary, we find that the number of tokens in the response correlates strongly with the choice of n. the correlation between n and median word count is 0.98. we show specific examples of the impacts of n in figure 4 . figure 1 : example of qap prompting -shows how the prompt triggers explanation of the question followed by an approach to solve the problem, detailed steps, finally leading to correct answer ## prompt impact in figure 1 , we highlight the structure of a standard qap output. first, the model breaks down the question in its own words and provides detailed analysis on each event. many of the steps highlighted in the explanation were shown in the calculation section. compared to the cot output, qap encourages more sophistication in its response and thus reaches the correct answer. ## analysis question difficulties based on baseline performance: within a given dataset, the difficulty of the individual question may vary. we propose a method to measure question difficulty based on performance with the baseline prompt. if the model can answer the problem correctly with the baseline prompt, then we consider the question to be easy; otherwise the question is hard. we analyze the performance of different prompts across "easy" and "hard" questions. impact of word counts on question difficulties: qap generates higher word counts for both "easy" and "hard" questions ( downsides of smaller qaps: despite high performance on strategyqa, qap25 performs poorly on arithmetic datasets (mostly sat and aqua) using gpt-3.5 turbo. due to a small value of n, the model outputs are unfinished responses (i.e. the model stops midway through its reasoning steps) (shown in figure 8 ). on sat math, 51% of responses were incomplete for qap25. on aqua, 19% of responses were incomplete for qap25. ## additional studies placement of the prompt: in this evaluation, we studied the impact of prompt placement on performance using gsm8k dataset. two options for prompt placement were considered: q_beginadding the prompt before the question, and q_end -adding the prompt after the question. both placements provided similar results on gpt-3.5 and gpt-4. results shown in the rest of the paper are based on q_end. no n constraint: to test the effectiveness of adding the value of n, we first examine the prompt with just the phrase: "explain this problem to me. then solve for the answer". however, the model does not explain the question completely and in most cases directly starts solving the question. its responses are no different than a response which used no given prompt. this shows that explicitly stating the minimum amount of words required is more likely to induce the model to explicitly generate an explanation of the question. ## related work in one-shot and few-shot prompting, the model is given one or more input/output examples which will serve as a demonstration for it to solve the problem using in-context learning @xcite . qap is a zero-shot prompt. in zero-shot 6 and chain of thought: chain-of-thought reasoning is a notable few-shot (zero-shot also exists @xcite example in which the model is shown how to express its reasoning steps @xcite . this approach was highly effective as the model would replicate these exemplars, and their accuracies improved drastically. cot encouraged the model to think step-by-step, and this concept would be a repeating theme among other zero-shot counterparts. tadb: among different variants of zero-shot cot, the tadb prompt @xcite was derived using an optimization objective to find instructions that would maximize task accuracy. the eventual prompt was "take a deep breath, and work on this problem step by step". tadb is an example of how the wording of a prompt can drastically impact responses. plan and solve prompting plus: another zeroshot prompt is plan-and-solve prompting @xcite . there were two versions to this prompt. the first simply asked the model devise a plan and solve step-by-step. the second version (ps+) extended the prompt by specifically asking the prompt to extract relevant variables and their corresponding numerals and to calculate intermediate results. we used ps+ on our experiments. one difference between ps+ and qap is that ps+ prompt is more specific to math datasets since it instructs to extract variables, intermediate results, etc., whereas qap is more general. also, ps+ prompts the model to understand the problem, but it is not clear if model should output anything specific to the question itself. in contrast, qap explicitly instructs the model to explain the problem in n words. question decomposition: question decomposition @xcite strategy causes the model to break down the question by creating sub-questions. the model answers each of these sub-questions and it ties together all the subanswers into a final answer. it considers two methods for decomposition, factored decomposition and cot decomposition. in factored decomposition each sub-question is answered in a separate context. cot decomposition is an intermediate between factored decomposition and cot. it enforces one context for sub-question, sub-answer and the answer to the original question. the analysis of question decomposition shows reduced bias and ignored reasoning, improves the faithfulness of a model-generated reasoning over cot while retaining the performance gains of cot. ## conclusion in this paper, we explored the approach of question-analysis prompting to improve llm accuracy across math and commonsense reasoning. the prompt focuses on how the model interprets the task given, and whether restating the question in its own words can further sophisticate its answer steps. the ability of this prompting method to perform well in diverse model types, tasks difficulty, and type of tasks seems promising. we plan to extend this work further by combining qap with other prompt strategies, applying decoding strategies and evaluating multi-modal tasks.