Evaluating the Deductive Competence of Large Language Models (2024)

Abstract: The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.

## introduction in this paper, we extend the previous research in several ways. first, we investigate the extent to which limited performance may be due to how the task was formatted. prior research has demonstrated that overall performance can vary according to how a particular task is formatted @xcite @xcite . research on analogy generation @xcite demonstrates that performance depends on the specific prompt given to the models. thus the inability of a model to solve one particular format of a task only provides a lower limit for assessing whether a model can successfully solve that task @xcite . second, limited performance on deductive reasoning may be due to how the researchers employed content familiarity, of direct relevance to the distribution of content in the training corpus. for familiar content, they tested several different types of problems including social rules (i.e. if peo-inference definition modus ponens @xmath0, p ∴ q deny the antecedent @xmath1, ¬p ∴ ¬q affirm the consequent @xmath2, q ∴ p modus tollens @xmath3, ¬q ∴ ¬p we expand the research base on the reasoning capabilities of llms by: 1) examining the role of specifically social-rules in reasoning about realistic content, 2) investigating the role of alternative presentation formats in deductive reasoning performance, and 3) expanding the set of candidate llms to evaluate potential algorithmic effects. our results show that social content does benefit llm performance, but not to the extent that might be expected based on a human sourced training corpus. while presentation formats do influence performance, they interact with content in a surprising (non-human) fashion. these findings are independent of the llm examined. ## evaluating deductive competence the wason selection task is a reasoning task from the cognitive science literature that evaluates deductive competence @xcite . participants are presented with a rule of the form if p, then q and four cards with p status on one side and q status on the other that correspond to the options p, ¬p, q, and ¬q. participants are asked to determine which card or cards must be flipped over to validate whether the rule holds for this set of cards. in the traditional abstract version of the task, participants are given rules about letters and numbers (i.e., if there is an odd number on one side of the card, there is a vowel on the other side of the card). the correct response requires the identification of two cards. typical human accuracy for these problems is 10-20% with common errors consistent with confirmation bias. in contrast, problems that deal with a social rule (i.e., if a person is drinking beer, they must be at least 21 years old) are easy to solve-most participants (70%+) correctly select both cards @xcite . there are four potential conditional inferences in task: modus ponens, denial of the antecedent, affirmation of the consequent, and modus tollens (evans, 2013). of these four inferences, only modus ponens and modus tollens are logically valid. these inferences and their logical forms are illustrated in the wason task makes a good candidate task for evaluating the reasoning performance of llms for several reasons. first, the task is relatively close to certain language modeling objectives. while the task involves a reasoning component, it can be formatted as a completion task, where the objective is to predict the answer given the problem text. this suggests that prior training, particularly for llms with high numbers of parameters that have been trained on large text corpora, should provide sufficient information for performing the wason task. moreover, the construction of the task minimizes the potential for confounds that may artificially inflate performance @xcite @xcite . previous work has demonstrated that high performance on some natural language inference tasks @xcite can be due to exploitable properties of the training data @xcite . the standardized format of the wason task allows for the creation of a large number of carefully constructed examples without the risk that some answers may be easily determined from the original prompt alone. second, the problem examines both straightforward and challenging aspects of deductive competence. as noted above, a correct answer to a wason problem involves two logical processes: modus ponens and modus tollens. results from the cognitive science literature indicate that these rules are not equally difficult-applying modus ponens is considerably easier than applying modus tollens. the vast majority of people correctly apply modus ponens, even for difficult problems (i.e., @xcite @xcite . in contrast, people fail to apply modus tollens, unless the problem has a particular form of semantic content associated with it. similarly, we might expect presentation format to assist llms. third, the way the task is constructed allows for careful examination of how problem content influences reasoning performance. because llms are built on co-occurring content in human-sourced corpora, they should benefit from problems that contain familiar relationships. this should be especially true for problems where the relationship between the antecedent and the consequent is highly familiar. for example, in the rule if a person is driving, they must have a driver's license, the antecedent driving a car, and the consequent driver's license have a familiar (and commonly occurring) relationship. in comparison, the antecedent and the consequent in a rule such as if a person is driving, they must have a book bag do not have the same familiar relationship. while it is likely that some problems may be more difficult than others (i.e., because some completions are more probable) we control for this experimentally. we create sets of problems where both arguments contain realistic content, but the relationship between them is unfamiliar (see appendix a). lastly, there is a large body of human performance literature on this task. this literature provides a comparison point for evaluating the performance of llms. ## experiments in this section, we discuss the conditions, task format, models, and evaluation metrics associated with our experiments. ## discussion we set out to expand the research base on evaluating the reasoning capabilities of llms with a classic experiment contrasting a priori conditions. we do replicate some effects found in the literature for human subjects. performance is higher for familiar problems than arbitrary ones and social rule have higher performance than non-social rules. however, we do not replicate the magnitude of the effects; social content does not benefit llm performance as might be expected based on training corpus content. we do find effects for different presentation formats; however, they do not improve performance. for shuffled problems, the specific presentation format does not make a difference. for realistic problems, we find presentation format does make a difference-the classic presentation format has the highest performance. however, our systematic, content and format controlled experimentation and performance measurement has also revealed a number of inexplicable interactions that appear to be consistent across different llms and are therefore independent of architecture. given the literature on human performance, an interaction between social rule status and content type is expected. however, many of the other interactions are not expected and are inconsistent with human performance (e.g., @xcite . we find some evidence that llms benefit from different types of presentation formats, (depending on the specific content of the problem), as might be expected from popular compensatory prompt engineering efforts. however, it is not immediately clear what types of information facilitate overall reasoning performance. this limits the ability to make general predictions about the conditions under which llm reasoning is accurate. in addition to content and presentation interactions, we find that llms do not pick antecedent cards at the same rate that human subjects do. moreover, this behavior is influenced by the task condition-llms are less likely to select antecedent cards for arbitrary and realistic problems than for social rules. overall low performance is particularly surprising for realistic social and non-social rules as the relationships for solving these problems are plausibly available in the training data of llms. yet we find that all llms diverge from documented human performance. overall performance is also remarkably consistent across models despite different training data, objectives, and model architectures. moreover, we find that the interaction results are also independent of the llm examined. some consistency between llms is to be expected, given that llms are trained on human-generated text corpora. however, the commonalities are not consistent with human behavior. this suggests a common yet surprising emergent reasoning bias without any apparent adap-tive benefit. fine-tuning the models for this task would likely improve task performance. we did not fine-tune the models for several reasons. first, the wason task is intended to be a general task for evaluating reasoning performance. these tap into general knowledge and a set of reasoning skills that transfers to new tasks. thus, the position that networks specifically trained on large reasoning task corpora is the best way to evaluate the reasoning performance of models is questionable. this is particularly true given that models often have high performance on the specific training task and limited performance on related reasoning tasks @xcite . second, several researchers have proposed that the wason task can be solved via linguistic and real world knowledge @xcite @xcite . human participants achieve high performance on problems that deal with familiar social rules with no experience with the task, using prior knowledge. however, the knowledge that this task requires is plausibly available in the training data for llms. the words used in this task are all common english words. moreover, many of the relationships between items are plausibly available in training text, particularly for problems that deal with familiar social rules. yet, performance remained quite low. previous work has proposed that ideal tasks for evaluating the reasoning of computational algorithms are those that do not require task-specific training @xcite . we concur with this position and suggest that the wason task is an ideal task in this regard. ## conclusion despite substantial performance improvements on standard benchmark datasets, existing llms have considerable room for improvement with regards to many aspects of human intelligence @xcite . in these experiments, we specifically investigate two of these aspects: generalized performance on related tasks and generation of answers at the limits of available knowledge. overall, our results replicate some of the same patterns found in the cognitive science literature. however, performance remains poor with inexplicable interactions between problem content and our efforts to manipulate presentation format. llms are sensitive to different sets of task criteria than human subjects. these criteria are not predictable across conditions and suggest areas where the reasoning of llms is not consistent with that of human capability.