Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024)

Abstract: Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like \textit{supposition following} or \textit{chain construction} . Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model’s accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.

## introduction deductive reasoning, that is the process of drawing conclusions that logically follow from the information at hand, is an integral aspect of human cognition and plays a pivotal role in formulating sound and coherent arguments @xcite . take, for example, the following statements: if there is a blue marble in the box then there is a green marble in the box. there is a blue marble in the box. ## strategies in propositional reasoning propositional logic studies the relationships among statements (or propositions) and the methods for constructing logical arguments based on them @xcite . at the core of propositional logic are simple statements that can be combined through the use of logical connectives such as "not", "and", "or", and "if... then...", thereby forming more complex compound statements. conclusions are logically deduced, where the truth value of the propositions necessitates the truth of the conclusion. this form of logical reasoning allows us to construct sound arguments that are invariably true, given the truth value of the information provided. as such, propositional logic is fundamental to various disciplines, including science, mathematics, and philosophy, where it offers a structured approach to reasoning and argumentation. to gain insights into the inferential processes humans employ in propositional reasoning, @xcite conducted a series of experiments that study the behavior of participants during propositional reasoning. they formulated straightforward propositional logic problems with neutral content (the presence or absence of colored marbles in a box, similar to the problem illustrated in figure 1 ) and requested participants to articulate their thought processes while engaging with these problems. participants were permitted the use of paper and pencil for their workings. both their verbal explanations and written responses were meticulously recorded, transcribed and analyzed thereafter. @xcite discovered five strategies reasoners commonly utilize to navigate the problems, offering insights into their inferential mechanisms employed during propositional reasoning. in the following, we give a short description of each strategy (illustrated in figure 2 ). for more details and additional examples, we refer to the original study by @xcite . @xcite (to the left of the dashed vertical line) that human reasoners employ when solving tasks of propositional logic. each strategy is illustrated by a single example adopted from the transcribed recordings published by the original study. in addition, we provide an example of the ○ symbolic strategy occasionally encountered in llms (to the right of the dashed line). "iff" denotes a biconditional, while "xor" indicates an exclusive disjunction. ## experimental setup task overview. our task setup aligns with the experiment conducted by @xcite to allow for a fair comparison between the inferential strategies found in humans and those identified in llms. 2 in particular, we evaluate each model on the 12 problems of propositional logic suggested by @xcite (an overview of each problem can be found in figure 5 in the appendix). for each problem, models are presented with a set of statements (or premises) and must determine whether a given conclusion logically follows (for an example, see figure 1 ). eight out of 12 problems involve three premises and a conclusion, while the remaining four problems consist of four premises leading to a conclusion. all premises, as well as the conclusions resemble either biconditionals, exclusive disjunctions or conditionals. two problems (4 and 6) include a redundant first premise. all premises are stated such that two subsequent statements contain one proposition in common, except of two problems (11 and 12), which are arranged in a non-sequential manner. for half of the problems, the conclusions logically follow from the premises, whereas for the other half, they do not. to avoid the influence of external knowledge and ensure content neutrality, @xcite framed the problems around the presence of colored marbles in a box, with colors assigned randomly to each entity within a problem. language models. we aim to investigate various factors that might impact the inferential strategies displayed by llms. these factors include the type of model, its size, and the emphasis on alignment during training @xcite . therefore, we assess a total of five models, consisting of three prominent open-access model types: llama 2 @xcite with model sizes of 7b, 13b, and 70b, the recently released mistral-7b model @xcite , and zephyr-7b @xcite , an extension of mistral-7b with a focus on intent alignment through fine-tuning with ai feedback (aif). for our evaluations, we utilize the publicly accessible model weights from the hugging-face platform, specifically llama-2-chat-hf foot_1 (7b, 13b, and 70b), mistral-7b-instruct-v0.2, 4 and zephyr-7b-beta. 5 we consciously opt not to include proprietary models accessible via paid apis, despite their reported superior performance in reasoning tasks @xcite . this methodological choice reflects our commitment to promoting transparent and reproducible scientific research. note that in this work, we refer to the above models when using abbreviations such as llama-2, mistral-7b-instruct or zephyr-7b-β. ## results and analysis in this section, we present the results of our evaluation. we begin with a quantitative analysis of the inferential strategies employed by llms, as well as the logical validity of their reasoning. this is followed by a qualitative analysis providing a more in-depth examination of the models' reasoning. ## conclusion in this paper, we examine the inferential strategies employed by llms in solving problems of propositional logic. through a comprehensive evaluation of their reasoning behavior, we demonstrate that llms adopt strategies akin to those observed in human reasoners. our quantitative analysis reveals that the frequency with which a model adopts a specific strategy strongly depends on its type, size, and fine-tuning procedure. moreover, our analysis suggests that the accuracy of a model's final conclusions does not adequately capture its reasoning capabilities, underscoring the importance of a more sophisticated evaluation framework that includes the model's reasoning paths. we also provide a qualitative analysis of typical reasoning behaviors among models, pinpointing prevalent errors such as difficulties in understanding negations or recognizing all implications of a supposition.