A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024)

Abstract: The reasoning abilities of Large Language Models (LLMs) are becoming a central focus of study in NLP. In this paper, we consider the case of syllogistic reasoning, an area of deductive reasoning studied extensively in logic and cognitive psychology. Previous research has shown that pre-trained LLMs exhibit reasoning biases, such as content effects, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning. We contribute to this research line by systematically investigating the effects of chain-of-thought reasoning, in-context learning (ICL), and supervised fine-tuning (SFT) on syllogistic reasoning, considering syllogisms with conclusions that support or violate world knowledge and with multiple premises. Crucially, we go beyond the standard focus on accuracy, with an in-depth analysis of the conclusions generated by the models. Our results suggest that the behavior of pre-trained LLMs can be explained by heuristics studied in cognitive science and that both ICL and SFT improve model performance on valid inferences, although only the latter can mitigate most reasoning biases while being consistent.

## introduction soon after the first transformer-based models entered the nlp scene, @xcite discovered that transformers are "soft reasoners": they can be trained to emulate reasoning over natural language sentences. since then, transformer-based language models have made enormous progress. a core question of today's ai research is whether large language models (llms) have latently learned reasoning skills in addition to acquiring language proficiency. if this is the case, such skills could be elicited through suitable prompting; if not, other methods should be proposed to achieve computational models that can reason using language as the inference vehicle. one fundamental aspect of human reasoning is the ability to derive valid conclusions based on the form of the propositions involved, rather than their content @xcite . this deductive reasoning skill is crucial in our everyday problem-solving and decision-making experiences, and its presence in llms is a topic of ongoing debate. ognize invalid inferences -premises from which no valid conclusion follows. furthermore, llms struggle when multiple reasoning steps are required @xcite , but @xcite show that when prompted with cot and in-context learning (icl) examples, they manage to generalize to out-of-distribution logical rules. we systematically bring these insights together, by comparing different learning strategies and by focusing on classical aristotelian syllogisms, since they are an interesting test bed to understand whether an agent detaches form from meaning. given previous findings, we first study the performance of small-and medium-sized open-access models (pythia and llama) in zero-shot cot (zs-cot) settings, relative to that of humans. subsequently, we examine the impact of both icl and supervised fine-tuning (sft) and consider how these approaches generalize to multi-step reasoning. this is achieved using carefully designed datasets of believable and unbelievable syllogisms, which respect or contradict world knowledge, respectively, and of syllogisms with 3-and 4-premises. crucially, we go beyond the standard focus on accuracy: by looking at all conclusions the models generate we inspect their consistency and whether their answers are predicted by heuristic theories proposed in the cognitive science literature @xcite . our experiments reveal that: ## related work before the recent surge of general-purpose llms, reasoning in transformer language models was primarily studied in the field of natural language inference (nli), a long-standing challenge within the computational linguistic comunity @xcite . the most recent research mainly utilized encoder-only models like bert @xcite , roberta @xcite , and xl-net @xcite . these models approached nli as a classification task, using datasets such as snli @xcite and mnli @xcite for fine-tuning. the literature in this area focused on both inductive and deductive inferences, with deductive inferences further categorized into formally valid and materially valid inferences (for a recent survey see @xcite . with the emergence of larger and more powerful decoder-only llms, many nlp tasks no longer required specific architectures or training. instead, a single model could perform most tasks. this shift led to increased interest in studying more abstract reasoning skills of llms @xcite @xcite . it has been established that chain-of-thought (cot) prompting elicits some reasoning capability @xcite and that in-context learning (icl) further boosts such skills @xcite . in this paper, we focus on syllogisms because they are an interesting test-bed to study the interleaved role of content and form in deriving conclusions, and have been extensively studied in the cognitive science literature. @xcite introduced an assembled general logical reasoning evaluation benchmark (glore) and compared chatgpt, gpt-4, and humans against open access llms, such as llama @xcite . they show that instruction-tuning, supervised fine-tuning, incontext learning, and voting techniques increase llama's task accuracy. they also raise the posaffirmative negative a: all a are b e: no a are b i: some a are b o: some a are not b 1 2 3 4 p1: a-b b-a a-b b-a p2: b-c c-b c-b b-c p1: all b are a (a) p2: no c are b (e) c: some a are not c figure 2: the building blocks: moods (a, e, i, o) and figures (1-4). their combination determines the conclusion, as illustrated by the ae2 schema. the two papers that are most related to our work are @xcite and @xcite . @xcite challenge proprietary llms with discriminative reasoning tasks, including the classification of syllogistic inferences as valid or invalid, and show that larger models are more logical than smaller ones, and llms suffer from the content effect bias. we extend this result by looking at llms as generative models. @xcite evaluate the palm 2 family @xcite on a generative version of the syllogism task; specifically, they use zero-shot cot prompting @xcite to select which conclusion, among a list of candidates, is the one that logically follows from a pair of premises. they show that even the larger models make mistakes that mirror human reasoning biases. moreover, they show that llms have difficulty with invalid syllogisms: they rarely generate "nothing follows" even when it is presented as one of the multiple choices. we extend this analysis of zs cot models by systematically investigating the effects of cot reasoning, icl, and supervised fine-tuning (sft) on syllogistic reasoning, considering syllogisms with conclusions that support or violate world knowledge and with multiple premises. @xcite show that models with cot and icl generalize to rules not seen inside their context but fail in generalizing to longer proofs. we advance our understanding of the role of icl by focusing on the interplay between content and form and carrying out in-depth analysis of the conclusions. in summary, previous contributions have identified various important dimensions and separately addressed the impact of zero shot cot, icl, and sft. in this paper, we systematically address these dimensions, compare model and human performance, carry out an in-depth analysis of outputs, an analysis missing in the current literature. ## syllogisms syllogisms consist of two premises and a conclusion that share the structure "quantifier x are y". as summarized in figure 2 (left) such statements can be in four moods: either affirmative or negative, and either universal or existential based on the type of quantifier ("all", "some" or "some . . . not" and "no"); they are abbreviated using letters -affirmatives as "a" and "i", negatives as "e" and "o" from the latin "affirmo" and "nego", respectively. the order in which the nouns occur within the premises defines the figure; there are four possible figures marked by numbers (1-4). only one noun (represented in figure 2 by the b variable) is shared between the premises, the relation holding between the other two nouns (the a and c variables), if any, is derived from the premises. hence, syllogisms consist of a chain of predications linking the terms of the conclusion. such inference can be defined by the combination of moods and figures, which are referred to as "schemas" and abbreviated by naming the premise moods and the figure number. figure 2 exemplifies the ae2 schema. interestingly, schemas can be composed to obtain longer inferences (longer chains) and can be applied recursively, requiring multi-hop reasoning @xcite . for instance, to infer a conclusion for the premises in figure 1 (bottom), one should first derive an intermediate conclusion from two premises, e.g., p1 and p2, and then combine the intermediate conclusion with the remaining premise to derive the final conclusion. this gives rise to multiple paths since more intermediate conclusions are possible. since each premise can be in one of the four moods, there are 64 distinct pairs of premises (16 moods x 4 figures). we follow the psychology literature @xcite and consider 37 of these combinations as invalid, while the remaining 27 lead to at least one conclusion (for the full list see appendix a). humans exhibit a content effect bias, namely the tendency to consider an inference valid when the conclusion is believable and invalid otherwise (e.g., @xcite . in their meta-review, @xcite aggregate results from six independent studies run with students of different levels (from high school to university) and show that the overall patterns of responses are robust and reliable; for humans, some schemas are easier than others, e.g., on the ai2 and the ae2 schemas, they obtain 90% and 1% accuracy, respectively. finally, humans seem to process valid and invalid syllogisms differently and have difficulty deciding that no valid conclusion follows @xcite . ## learning strategies we aim to understand whether llms are equipped with latent reasoning abilities to extract syllogistic schemas from given examples or if they could achieve such ability through supervised learning. to this end, we compare in-context and supervised learning. given the premises, we frame the problem as a multiple-choice task (see section 5). baseline we take as baselines the zero-shot cot (zs-cot) setting @xcite . zero-shot cot is a two-stage prompting method that provides no task examples to a model. in the first stage, a model is given a task instruction and prompted to think through a series of reasoning steps before generating a response -typically by adding the phrase "let's think step by step" before its response. in the second stage, the model is asked to provide a final answer based on its own reasoning chain. we take pythia and llama pre-trained models, and compare them in zs-cot looking at the effect of model size and, when possible, of undergoing a further instruction-tuning phase. 2 in-context learning in this setting, there is no further training, hence no weight changes. a model is presented with k examples of the multiple-choice task. it has been claimed that the diversity of the icl examples increases the model's performance @xcite . hence, as illustrated in figure 3 , we evaluate the setting in which the test example belongs to the same schema as the k randomly chosen in-context examples (icl in ), or belongs to a different one (icl out ); we set @xmath0. ## results based on task-accuracy in this section, first, we look into zs cot accuracy, then we compare different learning strategies to address the question of whether llms can learn to rely only on the form to generate a conclusion, an issue which has not received a systematic analysis yet. ## beyond task accuracy so far our evaluation has focused on whether there is at least one correct conclusion among those generated. in this section, we identify what is behind the weaknesses of models, by analyzing in depth all conclusions generated. as a final analysis, we check whether the models have learned some heuristics. by inspecting human responses across a variety of experiments, @xcite analyze the heuristic theories proposed as potential theories of human reasoning, concluding that none of these theories are good models of human reasoning: a good model should predict both the correct and the wrong conclusions an agent draws from the premises. we use these theories as a tool to further inspect models. we focus on the atmosphere theory since it turns out to be the most insightful. 9 the theory claims that "reasoners tend to accept a conclusion that fits the mood of the premise". in other words, the conclusion is drawn considering @xcite . we compare model performance by reporting the overall accuracy on believable syllogisms as well as its breakdown into valid vs. invalid schemas: pythia models are well below human scores, while llama models reach or outperform humans on valid syllogisms but are unable to recognize invalid inferences. this weakness is mitigated by instruction tuning but comes at the price of decreasing performance on valid syllogisms. all llama models except 70b instruction tuned, suffer from content effect bias (the performance loss from the believable to the unbelievable set on valid syllogisms). * marks when the performance loss is not statistically significant based on χ 2 's test. on the right, spearman ρ correlation coefficient between the accuracy of humans and models on the valid schemas of the believable set. the human accuracies used to compute the coefficient are based on @xcite . in all our analysis we noticed distinct patterns for the two sft models, namely pythia struggles more with an unseen number of premises ( although the results show that llama sft does not adopt a superficial strategy like the atmosphere heuristic and is more consistent, we cannot exclude that these results may still be driven by a more sophisticated form of pattern matching rather than genuine learning of the correct inference rules. this hypothesis is supported by the results on previously unseen numbers of premises (see ## conclusion models that can reason using language as the inference vehicle, or "soft theorem provers" in the words of @xcite , have been the holy grail of ai since its inception. now that llms have achieved amazing language proficiency skills, the community is investigating their reasoning ability. we contribute to these efforts by focusing on syllogistic inferences. we systematically compared zs-cot, icl, and sft and analyzed the answers generated by the models in depth. to our knowledge, we are the first to use heuristic theories, such as the atmosphere theory, to study llms. this heuristic turns out to be a very good predictor for llama-3 8b zs-cot; therefore, we conjecture that the reason zero-shot llms rarely generate "nothing follows" is that they use the quantifiers in the premises as cues. the merit of icl has been advocated by @xcite . by going beyond task accuracy and inspecting all the conclusions models generate, we show that while icl boosts model performance on valid inferences, it does not eliminate content effects or the difficulty in handling invalid syllogisms. most importantly, it increases model inconsistency. on the other hand, fine-tuning the model on next-word prediction is effective for both the small-and medium-sized models we evaluated and allows the latter to approach ceiling performance while remaining consistent. @xcite argue that language and reasoning are dissociated, and to be enhanced with the latter, lms should go through a different learning paradigm or be equipped with additional modules. our results suggest that while content-based reasoning can be learned from pre-training, formbased reasoning needs appropriate training. we leave for the future the question of whether llms after sft on a controlled dataset, as the one we use, transfer the acquired reasoning ability to real-life scenarios with linguistically richer expressions.