Multi- L ogi E val: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024)

Abstract: As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multi-step reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types — propositional, first-order, and non-monotonic consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs such as GPT-4, ChatGPT, Gemini-Pro, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs.

## introduction the ability to perform multi-step reasoning -drawing conclusions from provided multiple premises -is a hallmark of human intelligence. recently, large language models (llms) such as @xcite have achieved impressive performance on a variety of language tasks that were previously thought to be exclusive to humans @xcite @xcite . however, the ability of these llms to perform multi-step logical reasoning over natural language remains underexplored, despite its various real-world applications @xcite . although several datasets have been proposed @xcite to evaluate the logical reasoning capabilities of llms, these datasets are limited in their scope by (1) evaluating simplistic single-step logical reasoning such as prontoqa @xcite and (2) evaluating multi-step logical reasoning, but only on a single type of logic and covering only a few logical inference rules as done in fo-lio @xcite and proofwriter @xcite . furthermore, there are only a few benchmarks, such as logicbench @xcite and boardgameqa @xcite , that cover reasoning such as non-monotonic which is closer to human-like reasoning. motivated by this, our work aims to bridge these gaps by creating a more comprehensive and logically complex eval-uation dataset by incorporating varying numbers of reasoning depths (i.e., multi-steps) to reach conclusions. in addition, past attempts have been made to evaluate multi-hop reasoning of language models @xcite . in contrast, our work systematically evaluates multi-hop logical reasoning over various inference rules and their combinations. to this end, we propose multi-logieval, a systematically created question-answering (qa) dataset covering multi-step logical reasoning across three different logic types: propositional logic (pl), first-order logic (fol), and non-monotonic (nm) reasoning. our objective is to present a preliminary analysis of the llms' ability to perform multi-step logical reasoning and demonstrate their failures even when performing simple reasoning. we believe that, regardless of whether such reasoning is available in some existing natural data (e.g., examinations), llms should do proper logical reasoning. thus, we systematically compiled data using various inference rules and varying numbers of reasoning depths. in particular, our proposed dataset provides ∼ 1.6k high-quality instances that cover 33 inference rules and reasoning patterns and more than 60 complex combinations of these inference rules with a different number of reasoning steps (1 ∼ 5). our choice of inference rules is further explained in section 3.1. to evaluate llms, we formulate a binary classification task in multi-logieval where the context represents a natural language story consisting of logical statements, and the models have to determine whether the story logically entails a conclusion given in the question. examples of instances are presented in we evaluate a range of llms, including gpt-4, chatgpt, gemini-pro, yi-34b @xcite , orca-2-13b @xcite , and mistral-7b @xcite on multi-logieval using zeroshot chain-of-thought (zero-shot-cot) prompting @xcite . the zero-shot cot approach allows us to determine llm's ability to do logical reasoning based on parametric knowledge (acquired during pre-training) since we can not ex- ## related work past attempts have been made to assess the logical reasoning ability of language models. for instance, logiqa @xcite and reclor @xcite evaluate diverse forms of logical reasoning by compiling multi-choice questions from standardized examinations, including multi-step reasoning. however, in contrast to our multi-logieval, these datasets involve mixed forms of reasoning and do not focus on assessing logical reasoning independently. past attempts have been made to create datasets focusing on logical reasoning @xcite . in terms of task formulation, our proposed dataset is similar to proofwriter @xcite , ruletaker @xcite , folio @xcite , prontoqa @xcite , ## multi-logieval in developing multi-logieval, we leverage the capabilities of llms while employing different methods to generate data for nm compared to pl and fol since the formulations for pl and fol differ from nm. in particular, our data creation process consists of two major stages: (i) generation of rule combination and (ii) generation of data instances. ## conclusions in this work, we introduced multi-logieval, a comprehensive multi-step logical reasoning benchmark consisting of three types of logic and over 60 combinations of inference rules. our approach utilized two-stage methodology to construct data instances for our benchmark consisting of ∼ 1.6k data instances with 1 ∼ 5 reasoning depth. we evaluated a range of llms, including gpt-4, chatgpt, gemini, yi, orca, and mistral on multi-logieval. experimental results revealed that these models struggle to perform logical reasoning, and their performance drops as the depth of logical reasoning increases (average accuracy of ∼ 68% at d 1 to ∼ 43% at d 5 ) for classical and non-classical logic. furthermore, we systematically analyzed the reasoning chain generated by llms at various depths and presented interesting findings. we hope that multi-logieval will facilitate future research in evaluating and enhancing the ability of existing and upcoming llms for multi-step logical reasoning.