Over-Reasoning and Redundant Calculation of Large Language Models (2024)

Abstract: Large language models (LLMs) can solve problems step-by-step.While this chain-of-thought (CoT) reasoning boosts LLMs’ performance, it is unclear if LLMs know when to use CoT and whether those CoT are always necessary to answer the question. This paper shows that LLMs tend to generate redundant calculations and reasoning on a manually constructed math QA dataset, GSM8K-Zero.GSM8K-Zero is constructed such that the questions can be answered without any calculations, but LLMs, including Llama-2 models and Claude-2, tend to generate lengthy and unnecessary calculations to answer the questions.We also conduct experiments to explain why LLMs generate redundant calculations and reasonings.

## introduction llms have shown impressive performance on various tasks @xcite @xcite , and cot reasonings @xcite further boost the llms' performance. to make llms better learn to think step-by-step, it is common to enhance the cot ability of llms by adding training samples that contain cot reasoning in instruction tuning @xcite . while step-by-step reasoning is useful, it lengthens the llm's output. longer outputs imply longer inference latency and cost more money on commercial llms. moreover, if llms give long answers to questions that can be easily solved without complex reasoning, this may confuse the users and make them falsely believe that the questions are difficult and must be solved step-by-step. it is thus more desirable that llm only triggers cot reasoning ability when necessary, but it is unclear if a deep-sea monster rises from the waters once every hundred years to feast on a ship and sate its hunger. over three hundred years, it has consumed 847 people. ships have been built larger over time, so each new ship has twice as many people as the last ship. how many people were on the ship the monster ate in the first hundred years? how many people has it consumed over three hundred years? (a) question in gsm8k-zero to calculate the total number of people the deep-sea monster has consumed over three hundred years, we can sum up the number of people on each ship the monster has feasted on. the rst ship had 1 person (since it has been a hundred years), the second ship had 2 people (since the new ship has twice as many people as the last ship), the third ship had 4 people, and so on. (...) the formula is as follows: s @xmath0...) therefore, the deep-sea monster has consumed approximately 2.03 x 10 90 people over three hundred years. llms know when to think step-by-step and only generate step-by-step reasoning when necessary. prior works on cot mostly focus on the correctness of the cot reasoning steps or whether the reasoning steps are faithful to the question and support the final answer @xcite . however, no past study focuses on the redundancy in cot reasonings. as an initiative to study the redundancy of llm outputs, we aim to understand the following research question: does llm generate redundant reasonings when they clearly need not do so? to study this question, we construct a math qa dataset, gsm8k-zero, which contains trivial questions that can be answered without any calculations and reasoning. using this curated dataset, we can define the redundancy of output from llms. we evaluate seven llms trained with reinforcement learning with human feedback (rlhf) @xcite , and we find that llms tend to generate redundant calculations that complicate the responses and sometimes lead to the wrong answer. to ex-plain our observation, we show that gpt-4 (ope-nai, 2023) and chatgpt (openai, 2022), which are widely used in gathering the preference data for training a reward model in rlhf @xcite , show a strong preference towards long answers that contain redundant calculations, even if the long answers are incorrect. our contributions are summarized as follows: ## experiments we test llms on gsm8k-zero in zero-shot, as zero-shot inference closely mirrors most users' practical use of llms-as-assistants. instead of leveraging advanced prompting techniques like zero-shot cot @xcite or plan-and-solve @xcite , we present a single question to the llm and take its response. for each question, we sample one response from the llm. in our preliminary experiments, we find the observations in our paper are robust toward the hyperparameters used for sampling outputs from llms. our evaluation encompasses proprietary llms, such as gpt-4, chatgpt, claude-2 @xcite , and palm (text-bison-001) @xcite , and open-source ones like llama-2-chat models of different sizes @xcite . we assess llms' performance on gsm8k-zero using two metrics: (1) redundancy: determined by the percentage of llm answers containing numerical operators like ×, +, and =. (2) accuracy: accuracy measures how often the llm's answer, extracted using a regular expression, aligns with the gsm8k-zero ground truth. ## conclusion in this paper, we construct gsm8k-zero to illustrate the redundancy in the output from llms. we show that llms tend to generate redundant calculations and unnecessary reasoning, sometimes leading to a wrong answer. we reveal that llms may not differentiate questions requiring step-bystep reasoning from simpler ones, suggesting a possible direction for improving llms. to explain our observation, we use proxy rms and find that these models prefer lengthy answers even if they are wrong. through this paper, we hope future researchers can focus more on the redundancy of the outputs of llms and develop training techniques to teach llms when to think step-by-step.