Categorical Syllogisms Revisited: A Review of the Logical Reasoning Abilities of LLM s for Analyzing Categorical Syllogisms (2024)

Abstract: There has been a huge number of benchmarks proposed to evaluate how large language models (LLMs) behave for logic inference tasks. However, it remains an open question how to properly evaluate this ability. In this paper, we provide a systematic overview of prior works on the logical reasoning ability of LLMs for analyzing categorical syllogisms. We first investigate all the possible variations for categorical syllogisms from a purely logical perspective and then examine the underlying configurations (i.e., mood and figure) tested by existing datasets. Our results indicate that compared to template-based synthetic datasets, crowdsourcing approaches normally sacrifice the coverage of configurations (i.e., mood and figure) of categorical syllogisms for more language variations, thus bringing challenges to fully testing LLMs under different situations. We then summarize the findings and observations for the performance of LLMs to infer the validity of syllogisms from the current literature. The error rate breakdown analyses suggest that the interpretation of quantifiers seems to be the current bottleneck that limits the performance of the LLMs and is thus worth more attention. Finally, we discuss several points that might be worth considering when researchers plan to release categorical syllogism datasets. We hope our work will provide a timely review of the current literature regarding categorical syllogisms, and motivate more interdisciplinary research between communities, specifically computational linguists and logicians.

## introduction large language models (llms) have achieved remarkable performance on a variety of tasks @xcite @xcite . over the years, a large number of benchmarks have been proposed that try to evaluate the different abilities of llms, many of which are designed for measuring logical reasoning ability using a variety of tasks. @xcite propose an argument reading comprehension task to test deductive reasoning. clutrr @xcite tests inductive reasoning capabilities by requiring to infer kinship relations between characters in short stories. reclor @xcite , mmlu @xcite , and logiqa @xcite contain multiple-choice reading comprehension questions to evaluate diverse forms of logical reasoning. datasets such as syllobase @xcite and folio @xcite require llms to conduct inferences using syllogism logic or first-order logic. among these datasets, many consist of questions that are directly taken from exams. for example, mmlu @xcite ) contains practice questions from tests such as the graduate record examination (gre), and reclor @xcite ## a concise introduction to syllogisms in this section, we provide a brief introduction to categorical syllogisms from a logician's perspective. we will show in sections 3.2 and 4.3 that these preparations will help us evaluate current syllogism datasets and better understand the bottleneck of the performance of llms. ## conclusion this work tries to address the question of whether current proposed benchmarks can evaluate logical reasoning abilities accurately and thoroughly. we choose categorical syllogism as our main focus, since this logical system has been extensively studied by logicians and has many nice properties, such as a finite number of possible cases, and automated ways of solving it. a categorical syllogism is also arguably the most basic building block for any other more complex reasonings. we draw the inspirations from how logicians analyze categorical syllogisms and construct a list of variations that should be covered by benchmarks. our results show that there is no single dataset that properly covers all possible situations. we also summarize the current progress made in judging the validity of the categorical syllogisms. our findings highlight the importance of correctly interpreting different quantifiers. finally, we provide a discussion of several points that might be worth considering when researchers plan on the future release of categorical syllogism datasets.