Evaluating Large Language Models with N eu BAROCO : Syllogistic Reasoning Ability and Human-like Biases (2023)

Abstract: This paper investigates whether current large language models exhibit biases in logical reasoning, similar to humans. Specifically, we focus on syllogistic reasoning, a well-studied form of inference in the cognitive science of human deduction. To facilitate our analysis, we introduce a dataset called NeuBAROCO, originally designed for psychological experiments that assess human logical abilities in syllogistic reasoning. The dataset consists of syllogistic inferences in both English and Japanese. We examine three types of biases observed in human syllogistic reasoning: belief biases, conversion errors, and atmosphere effects. Our findings demonstrate that current large language models struggle more with problems involving these three types of biases.

## introduction syllogistic inferences and their various variants have been extensively studied since prior analytics by aristotle in the 4th century bc. while the aristotelean syllogism is a small part of the predicate logic and a limited inference system when compared to, for example, the formal system of logical inference rules, there has been recently a revival movement of aristotelean syllogism and its variants, including natural logic @xcite @xcite . this renewed attention arises from the perspective of viewing syllogistic inferences as a "natural" inference rule applicable to our everyday reasoning in ordinary language. not only is there a re-evaluation of the significance of syllogistic inferences and their variants in relation to their usefulness in ordinary language, but they are also considered as a benchmark for various inference studies in different disciplines. for example, cognitive psychological studies of logical inferences @xcite , diagrammatic logical inference studies @xcite , neuroscientific studies of logical inferences @xcite , all draw upon syllogistic reasoning as a point of reference. on the other hand, the recent developments of deeplearning-based ai-tools of natural languages, in particular, the state-of-the art large language models (llms), including bert @xcite and gpt @xcite , are remarkable. these tools hold the potential to be useful for logical inferences. however, there is still a need for further accumulation of studies on the use of ai models for logical inferences in natural language. in this paper, we explore the potential of llms for performing logical inferences, with a specific focus on using syllogistic inferences as a benchmark. we present the neubaroco dataset, a new dataset consisting of syllogistic inferences in both english and japanese. the dataset is derived from the question collection baroco @xcite , which has been used in various studies in japan to evaluate human syllogistic reasoning abilities. the field of cognitive science of human reasoning has not yet been fully integrated with the recent advancements in ai, despite its potential to provide valuable insights for ai inference research. considering the significant attention given to biases in studies of logical inference within cognitive psychology and cognitive science @xcite , our primary focus lies in assessing whether llms exhibit the biases observed in human logical inferences. we focus on three types of human biases that have been studied in cognitive science, namely, belief biases, conversion errors, and atmosphere effects (see section 4). we explore the extent to which currently available llms for natural language inference can effectively address syllogistic inferences that are susceptible to errors resulting from these biases. the contributions of the paper are summarized as follows. the dataset will be made publicly available for research on human and machine understanding of logical inferences. ## human-like biases based on the above classification of the types of syllogistic inferences and sentences, we examine three types of human-like biases that can cause reasoning errors: belief biases, conversion errors, and atmosphere effects. we annotated information to each inference in the dataset to make explicit which inferences are misjudged by these biases. ## conclusion in this paper, we investigated syllogistic reasoning ability of current large language models in focusing on human-like biases that have been studied in the context of cognitive science of human reasoning. the experiments indicated that the state-of-the-art models fail for problems where errors are caused by various human-like biases, and that there is large room for improvement in deductive reasoning capabilities of large language model. among other things, our results on conversion errors suggest the importance of distinguishing the problems of interpreting sentences (in particular, interpreting quantifiers and negation) from the problem of performing logical inferences. for conversion cases, there is abundant room for discussion on at which level the models mistook. further inquiry into this issue could provide insight into a better understanding of the behavior of neural models. there remain many issues to be addressed. first, although we tested the models with no solved examples, experiments on few-shot learning will be insightful. @xcite reported that the models performed syllogistic reasoning better with few-shot learning, whether or not the content of the inferences were consistent with common-sense beliefs. in addition, we suppose that the performances can be different depending on the wording in instructions. further research will contribute to improvement of instructions. second, we showed that the models performed worse in the cases in which humans have troubles because of some well-known human biases. it is left for future work to make more detailed comparisons between humans and neutral models, which is a promising research direction since the original baroco data on human reasoning ability is available. finally, it is interesting to consider extended forms of natural logic inferences other than basic syllogisms, including relational syllogism with transitive verbs and comparatives and those inferences with generalized quantifiers such as most. it is left for future work to examine whether the models show similar biases for such extended syllogism inferences. and mkk477j.