D iv L ogic E val: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models (2025)

Abstract: Logic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs). Popular benchmarks may entangle multiple reasoning skills and thus provide unfaithful evaluations on the logic reasoning skill. Meanwhile, existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from the distribution of an ideal logic reasoning benchmark, which may lead to biased evaluation results. This paper thereby proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way. To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in LLMs. Through experiments, we demonstrate the extent to which logical reasoning is required to answer the questions in DivLogicEval and compare the performance of different popular LLMs in conducting logical reasoning.

## introduction large language models (llms) have been evaluated across diverse reasoning capabilities, including narrative reasoning @xcite , mathematical reasoning @xcite , inductive reasoning @xcite , and logical reasoning @xcite . among these, logical reasoning has long been regarded as a key indicator of human intelligence, frequently employed in contexts such as academic admissions, employment screening, and civil service recruitment. many globally recognized standardized tests, such as the lsat, gmat, civil service examinations, and general aptitude tests, include a substantial number of logic-based questions. in fact, various reasoning (e.g., commonsense reasoning and logic reasoning) skills may entangle in these benchmarks and some other reasoning skills can make non-negligible contributions to solving the task. for example, as shown in experiments (see §3.1), by simply reducing the model's ability to utilize logical reasoning through prompting, llm surprisingly yields better performance on re-clor and logiqa. consequently, such benchmarks may overestimate the logic reasoning abilities of llms. meanwhile, some efforts have been made to create datasets based on classical logic @xcite @xcite . for example, folio is equipped with human annotations of classical logic @xcite but it only contains about two hundred instances due to annotation cost. other datasets are synthetically generated by predefined templates over classical logic expressions @xcite @xcite . unfortunately, the language variety within these benchmarks is limited due to the high validate by logic solver ## effectiveness of divlogiceval divlogiceval effectively decreases the dependency on other reasoning abilities and mitigates the risk of distribution bias by providing a more diverse dataset. ## conclusion in this paper, we present divlogiceval benchmark, which is specifically designed to evaluate current llms in logical reasoning. divlogiceval includes a synthetic dataset as well as a new evaluation metric. the benefits of this dataset are two-fold: it reduces its reliance on other reasoning abilities and therefore provides more faithful evaluation in logic reasoning compared with popular benchmarks such reclor and logiqa; it is better in language diversity compared to existing classical logic benchmarks and thus alleviates the potential risk of distribution deviation. in addition, our proposed evaluation metric is able to alleviate some shortcomings suffered by existing metrics for evaluating llms. to investigate the logical reasoning abilities of different llms, a comparative analysis is performed. the results highlight both the strengths and limitations of current llms in logical reasoning, paving the way for more comprehensive investigations into their reasoning capabilities. acknowledgement this work has been made possible by a research impact fund project (rif r6003-21) and a general research fund project (grf 16203224) funded by the research grants council (rgc) of the hong kong government.