Adaption-of-Thought: Learning Question Difficulty Improves Large Language Models for Reasoning (2024)

Abstract: Large language models (LLMs) have shown excellent capability for solving reasoning problems. Existing approaches do not differentiate the question difficulty when designing prompting methods for them. Clearly, a simple method cannot elicit sufficient knowledge from LLMs to answer a hard question. Meanwhile, a sophisticated one will force the LLM to generate redundant or even inaccurate intermediate steps toward a simple question. Consequently, the performance of existing methods fluctuates among various questions.In this work, we propose Adaption-of-Thought (AdoT), an adaptive method to improve LLMs for the reasoning problem, which first measures the question difficulty and then tailors demonstration set construction and difficulty-adapted retrieval strategies for the adaptive demonstration construction. Experimental results on three reasoning tasks prove the superiority of our proposed method, showing an absolute improvement of up to 5.5% on arithmetic reasoning, 7.4% on symbolic reasoning, and 2.3% on commonsense reasoning. Our codes and implementation details are available at: https://github.com/NLPGM/AdoT

## introduction reasoning refers to the drawing of inferences through the use of reason. solving reasoning tasks requires substantial knowledge and powerful analytical capabilities, which can narrow the gap between human and machine intelligence @xcite . previous small language models (slms) based methods often adopt the fine-tuning technique to enhance their performance. however, the reasoning ability of slms is naturally limited by the size of their parameters and fine-tuning requires abundant labor-intensive labeled data. recently, large language models (llms) @xcite @xcite have demonstrated significant efficacy in various tasks. reasoning by prompting llms has also attracted wide attention. typical methods include automatic demonstration construction @xcite , rationale refinement @xcite , reasoning in code format @xcite , iteratively reasoning @xcite , problem decomposition @xcite , problem planing @xcite , and prompting ensemble @xcite . despite the effectiveness of these methods, it is interesting that none of them can consistently perform the best on most datasets on three reasoning tasks as shown in overall, php and complex-cot perform much better than standard. however, the pie charts in fig. 1 (a) show that there are up to 13.4% and 12.6% questions to which standard can answer correctly but php and complex-cot cannot. a closer look reveals that the reason can be the mismatch between the question difficulty and the complexity of the method. given the easy question in fig. 1 (b), php iterates four times and complex-cot builds the highly complex demonstration to interact with the llm. the generated redundant information misleads the model to produce wrong answers. meanwhile, standard, which only contains the sample questions and their final answer in its demonstrations, gets the correct answer. on the contrary, given the hard question in fig. 1 (c ), php and complex-cot give correct answers, but stan-dard fails since its simple demonstration prevents it from drawing out sufficient knowledge from the llm. to address the mismatch problem, we propose a novel adaption-of-thought (adot) method to adaptively adjust the complexity of the prompting method to the question difficulty. specifically, we first present a difficulty measuring approach for questions which computes the syntactic and semantic complexity of their rationales. we then propose a demonstration set construction and a difficultyadapted retrieval strategy to adaptively construct reasonable demonstrations based on the difficulty of the questions. extensive experimental results prove that our proposed adot method surpasses competitive baselines on all tasks. in summary, the main contributions of our work are as follows. ## related work this section reviews the literature on reasoning. reasoning by fine-tuning slms. slms-based reasoning methods @xcite need to fine-tune on task-oriented or domain-specific data to enhance their performance. for example, @xcite trained @xcite on both implicit pre-trained knowledge and explicit free-text statements to symbolic reasoning. @xcite fine-tuned the gpt-2 @xcite to generate full step-by-step solutions to arithmetic reasoning. though these approaches have shown better performance than traditional rule-based @xcite , symbolic-based @xcite , and statisticalbased @xcite methods, their reasoning ability has been limited by the size of slms. moreover, the fine-tuning of slms requires high-quality training data, which is quite labor-intensive. reasoning by prompting llms. llms-based approaches have become prevalent in recent years and prompting is the main way to interact with llms. many prompting methods have been proposed for reasoning problems @xcite @xcite @xcite . the first attempt was made by @xcite , which developed a zero-shot prompting method by adding a natural language description of the task in the prompt. notably, chain-of-thought (cot) prompting @xcite , which performed the multi-step reasoning through step-by-step rationale examples, achieved excellent performances in reasoning. @xcite found that adding instructions to encourage llms to automatically reason through some intermediate processes also contribute to improving performance. subsequently, a series of studies optimized the cot prompting by decomposing the problem or refining the rationale and demonstration from various perspectives @xcite @xcite @xcite . while these llms-based methods have made impressive progress, none of them has taken the difficulty of the question into consideration, leading to a mismatch between the problem and the solution. in contrast, our method can automatically adapt the demonstration to questions with varying difficulty. step 1. measuring question difficulty step 2. adaptively building a demonstration set concatenate easy hard normal few-shot-cot demonstration coreference resolution step decomposition optimized demonstrations difficulty section judgement step 3. adaptively retrieving demonstration target question： what is the num of … ？ difficulty: 132 sample question 2 sample question 6 sample rationale 2 sample rationale 6 demonstration target question difficulty -adapted retrieval final rationale and answer coefficient of variation of difficulty hard normal easy difficulty section partitioning demonstration set randomly select n sample questions sample question difficulty：216 sample question 1 sample rationale difficulty rationale i rationale i target question i syntatic complexity semantic complexity difficulty target question 2 rationale 1 rationale 2 few-shot-cot by llms target question 1 sample rationale 1 ## adaption-of-thought method this section presents our proposed method. ## conclusion in this paper, we find the mismatch between the question difficulty and the prompting complexity will hinder eliciting proper knowledge from llms. to solve this problem, we propose a novel adot method to adaptively adjust the complexity of the promoting method based on the difficulty of the target question. we first propose a difficulty measurement strategy. then, we construct a demonstration set through the optimized demonstrations. finally, we present a difficulty-adapted retrieval strategy to adaptively retrieve reasonable demonstrations based on the difficulty of the target question. a series of experiments prove that the mismatch problem is critical in reasoning and our method is effective on this problem. consequently, our adot method achieves a new state-of-the-art performance on various reasoning tasks.