Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning (2023)

Abstract: Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. To better facilitate research on syllogistic reasoning, we develop a benchmark called SylloBase that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks. We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet. To improve our dataset’s naturalness and challenge, we apply GPT-3 to paraphrase the template-based data and further manually rewrite 1,000 samples as the test set. State-of-the-art pre-trained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77% by RoBERTa on SylloBase, which indicates the great challenge of learning diverse syllogistic reasoning types on SylloBase. Our datasets are released at https://github.com/casually-PYlearner/SYLLOBASE .

## introduction reasoning, as a typical way for human beings to obtain new knowledge and understand the world, is also an ultimate goal of artificial intelligence @xcite . reasoning skills, i.e., examine, analyze, and critically evaluate arguments as they occur in ordinary language, have been required by many natural language processing tasks, such as machine reading comprehension @xcite , open-domain question answering @xcite , and text gener-* corresponding author. ation @xcite . 1 according to different mental processes, reasoning can be categorized as deductive, inductive, abductive, etc @xcite . in piaget's theory of cognitive development @xcite , these logical reasoning processes are necessary to manipulate information, which is required to use language and acquire knowledge. therefore, the study of logical reasoning is worthy of our attention because it is so prevalent and essential in our daily lives. in this study, we focus on syllogism, which is a typical form of reasoning and has been studied for a long time (it was initially defined in aristotle's logical treatises organon, composed around 350 bce). as shown in to support further study on syllogistic reasoning, in this work, we build a new natural language benchmark-syllobase, which has the following features (some examples are shown in the experimental results indicate that there is a great deal of room for improvement in the syllogistic reasoning capabilities of existing models. our additional experiments demonstrate the efficacy of transferring knowledge learned from our automatically constructed syllogism to actual human reasoning. the left side of figure 1 shows a well-known categorical syllogism about "socrates is mortal". we can see a categorical syllogism usually contains two premises and a conclusion. a common term (e.g., "human") links two premises, and the premises respectively define the relationship between "human" and "mortal" or "socrates". the reasoning process is to draw a conclusion based on the premises. a syllogism can also be described by a pattern, as shown in the middle side of figure 1 . ## data construction our target is to develop a large-scale benchmark and support research on several typical kinds of syllogistic reasoning. it is straightforward to collect data through human annotation, as most existing datasets have explored @xcite . however, this method is impracticable for obtaining large-scale data due to the high cost of human annotation. therefore, we propose constructing a dataset automatically from existing knowledge bases and man-ually rewriting 1,000 samples as the test set. ## conclusion in this work, we built a large-scale benchmark for natural language syllogistic reasoning. it covers five types of syllogism. the data were automatically constructed from knowledge bases by our proposed construction methods. to evaluate the models' performance on real human syllogism, we manually rewrite 1,000 samples as the test set. experiments show that syllogistic reasoning is a very challenging task for existing pre-trained language models. moreover, our further study indicates that existing models are even farther from tackling syllogistic reasoning in real scenarios.