Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked? (2025)

Abstract: Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study answers an unexplored question about the impact of different question types on LLM accuracy on reasoning tasks. We investigate the performance of five LLMs on three different types of questions using quantitative and deductive reasoning tasks. The performance metrics include accuracy in the reasoning steps and choosing the final answer. Key Findings: (1) Significant differences exist in LLM performance across different question types. (2) Reasoning accuracy does not necessarily correlate with the final selection accuracy. (3) The number of options and the choice of words, influence LLM performance.

## introduction performance of large language models (llms) on reasoning tasks has been extensively studied using diverse benchmarks @xcite @xcite . several benchmarks use short answer questions (saqs) where the llms generate a response, often accompanied by an explanation. multiple-choice questions (mcqs) are widely used because they allow for simple assessment @xcite . llms select the correct option among the given options. for true or false questions (tfqs), llms judge if the given statement/question is true or false. factors that may influence llm performance include the number of options, the sensitivity of word choices in mcqs, and whether "true" or "false" is the correct answer in tfqs. assessing llm performance can be challenging due to the diverse question types. saqs have the advantage of analyzing reasoning steps with clear answers. for mcqs, llms need to understand the options correctly and choose the right option. llms may overlook the reasoning steps. tfqs require llms to understand true or false accurately. motivated by these challenges, we evaluate llm performance on reasoning questions across different question types on two reasoning tasks, quantitative reasoning task and a deductive reasoning task. quantitative reasoning datasets were evaluated to benchmark multiple existing llms @xcite @xcite , and it contains arithmetic calculation. we also chose deductive reasoning tasks to evaluate llms' logical reasoning without arithmetic calculations. our study focuses on assessing the suitability of these question types for benchmarking purposes and gaining insights into the behavior of llms. we propose final selection accuracy and reasoning accuracy for performance assessment. final selection accuracy evaluates only the final answer in the output, such as the selected option in mcqs or tfqs. reasoning accuracy assesses the correctness of the reasoning steps leading to the final answer. checking reasoning accuracy is time-consuming since it cannot be automated. consequently, we introduce the following research questions: contribution #1: the aforementioned research questions and the experimental design to answer the questions. we carefully designed the options for mcqs and tfqs. to our best knowledge, rq1, rq2, rq3, and rq5 have not been investigated. rq4 was partially investigated in @xcite but with the following differences: (1) different numbers of options, (2) the wrong options in our mcqs covering a wide range of mistakes, and (3) the inclusion of the "something else" option that enlarges the solution space in reasoning in our mcqs. contribution #2: new key findings from evaluations of five llm models, two closed-source llms, and three open-source llms on quantitative reasoning problems selected from the gsm8k dataset @xcite and deductive reasoning problems selected from the babi dataset @xcite . we will share our code and benchmark publicly foot_0 . some key findings are as follows. (1) there are more statistically significant differences in the final accuracy across different question types compared to the reasoning accuracy. (2) reasoning accuracy does not necessarily correlate with final selection accuracy. (3) llms perform better with "true" as correct answers than "false" in tfqs. for mcqs, the llm performances vary significantly based on the number of questions. the insight is potentially useful to guide future benchmark development and improve llm performance. ## related work diverse benchmarks @xcite were used to evaluate llms @xcite @xcite and prompting methods @xcite . several datasets @xcite @xcite include mcqs @xcite . moreover, factors influencing performance on mcqs have been studied, for instance, selection biases by order of options @xcite @xcite , and types of token options @xcite . @xcite found that llms can solve mcqs without knowing the question after some few-shot examples. @xcite claims that the llm performance of mcqs is better than those of long-form generation questions on the care-mi dataset. they do not find a significant difference in the different number of options for mcqs. in their study, the number of options varies from 2 to 4 without "something else" as a possible answer. our mcqs contain 5 options and 11 options with "something else" as the final answer. ## benchmark construction our goals for designing the benchmark include (1) ensuring that multiple-choice and true-or-false questions are as difficult as short-answer questions in terms of reasoning; (2) evaluating llm accuracy in both choosing the final answers and its reasoning steps; and (3) answering the three research questions while avoiding the bias issues mentioned in the related work. ## rq2: do question types impact the reasoning accuracy of llm? ## rq3: what are the patterns of the incorrect outputs by llms on different question types? this analysis focuses only on incorrect output cases where either the final selection, the reasoning, or both are incorrect. figure 2 shows examples of different patterns. our analysis reveals distinct patterns of mistakes for different types of questions. figure 3 shows statistics of the patterns by different models on 5-option mcqs (u) and tfq (question-false). ## rq5: what factors of tfqs influence llms' performance? ## conclusions and future work llm performance on different question types fluctuates. our experimental results show that the reasoning accuracy is not always correlated with the final selection accuracy for multiple-choice ques-tions and true or false questions. to improve llm performance on these question types, it is important to improve both reasoning accuracy and the accuracy of selecting the final answer. the trends in significant differences vary across different reasoning tasks. for mcqs, factors such as the position of correct answers, the number of options, and the selection of words in options impact llm performance. in tfqs, factors such as type of instruction, whether "true" or "false" is the correct answer, and the selection of words influence llm performance. this paper presents a comprehensive evaluation of the impact of different question types on llm performance, but with the following limitations. the experiments do not involve prompting methods or few-shot learning strategies. the proposed measure of reasoning accuracy does not account for accuracy at individual reasoning steps, nor does it consider how the number of reasoning steps affects performance. while deeper insights could be obtained through such a study, it would require substantially more manual labeling effort. lastly, due to computational constraints, the experiments were conducted on subsets of the two datasets. nevertheless, we conducted statistical tests to evaluate the impact of question types on llm performance.