Paper: LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
Aside from inference rules, there are additional correlated metadata that would be helpful to add and consider when comparing model performance across subsets.

Author response:
Yes, your understanding is indeed correct. The question with the answer ‘yes’ has the correct application of the rule, and the question with the answer ‘no’ has a conclusion that is not supported by context (rule). Furthermore, we agree that different variation types can be added as metadata for a better understanding of data. Thanks for this valuable suggestion, and we will incorporate it in the revised version.

Reviewer comment (Evaluation):
It is not clear what the reader gains from the fine-tuning experiment.

Author response:
Thank you for this suggestion. With our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. Thus, we consider these experiments valuable. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

Reviewer comment (Experiments):
In the description of the NL conversion, the authors present a verbose specification of "or" to distinguish it from "xor". The authors should comment on this unwieldy specification, clarify why simpler versions were insufficient, and indicate whether a simpler specification was attempted initially and resulted in low performance due to disambiguation issues. More information is needed here.

Author response:
Thank you for this comment. Yes, indeed we want to make an explicit distinction between ‘or’ and ‘xor’. We found that understanding the logical implication of 'or' when integrated into logical formulations posed challenges to both humans - our human evaluators, mostly graduate students - and models in terms of understanding it. So with the focus of the paper on the inference rules, we decided to make the desired meaning of `or’ explicit using the above construct, so that our human evaluations and annotations would be correct. See the below example from LogicBench(Aug) where the use of our template vs. the direct 'or' formulation:

```
Context: If Sam has not finished his homework, then he will not go to the party.

Contextual Template Question (from our dataset): "Based on the context, can we ascertain that at least one of the following must always be true: (a) Sam has finished his homework, and (b) He will not attend the party?"

Direct 'or' Question: "Can we conclude that Sam has finished his homework or he will not attend the party?"
```

From the example provided above, it becomes evident that a question using a direct 'or' can be challenging to interpret and may cause confusion. Hence, we choose to go with the more expressed formulation of ‘or’.

Reviewer comment (Evaluation):
Some clarifying information on the metric is required. The reviewer’s understanding is that each instance has 4 variants, with 1/4 showing the correct application of the rule (measured by A(Yes)), and 3/4 showing variants that cannot satisfy the conclusion (measured by A(No)). If this is incorrect, it should be clarified for readers. For the A(No) problems, it appears that some examples provide evidence that cannot prove the conclusion or its negation, while others provide evidence that can prove the negation. For example, in Figure 2 Stage 3, Variation 2 implies "not S1", whereas Variations 3 and 4 can neither prove "S1" nor "not S1". These patterns of variants should perhaps be noted differently in the dataset metadata, as certain audiences may care about the distinction between these error types.

Author response:
Yes, your understanding is indeed correct. The question with the answer ‘yes’ has the correct application of the rule, and the question with the answer ‘no’ has a conclusion that is not supported by context (rule). Furthermore, we agree that different variation types can be added as metadata for a better understanding of data. Thanks for this valuable suggestion, and we will incorporate it in the revised version.

Reviewer comment (Evaluation):
It is unclear what is gained from the fine-tuning experiments. The performance increases on other benchmarks are minimal, and space might be better spent clarifying benchmark details (such as those raised above) that reflect the core contribution of the paper. The fine-tuning section could be moved to the appendix unless other reviewers find it informative, in which case the contribution should be explained to the reader more explicitly.

Author response:
Thank you for this suggestion. With our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. Thus, we consider these experiments valuable. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

Reviewer comment (Experiments):
For example, whether the prompting method is the best approach to let the model speak out the reasoning steps that corresponds to the reasoning decision given the variances and limitations of generations of GPT-3/4.

Author response:
We note that the emergence of the prompt paradigm has shifted the assessment of LLMs, favoring zero-shot and few-shot evaluations over traditional supervised fine-tuning (SFT). Thus, our work also adapts a similar method and evaluates LLMs using Zero-shot-CoT, a standard practice for meaningful reasoning with LLMs. While evaluating LLMs such as GPT-3/4, we set the temperature hyper-parameter to 0 to control the variation in generation. Furthermore, we also conducted a manual qualitative analysis (section 4.3) to support our findings.

Reviewer comment (Experiments):
Are Yes and No questions balanced in the benchmark?

Author response:
No, they are not balanced. Thus, we presented a separate accuracy in Table 6, A(Yes) and A(No) to evaluate model performance. We updated this in the paper (Section 3.3).

Reviewer comment (Evaluation):
The authors indicate that the LogicQA and ReClor include richer reasoning formats, but will it necessarily undermine the impacts of finetuning on LogicBench?

Author response:
LogicBench is designed to evaluate LLMs on mainly single-step reasoning across a range of inference rules and show that they struggle with even single-step reasoning for complex inference rules. Furthermore, errors made during the initial single-step reasoning can be propagated through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios. Thus, teaching models first to do better single-step reasoning via different approaches (fine-tuning in our case) and then utilizing them to perform reasoning over complex and longer formats (LogiQA, ReClor) can improve their performance (as shown in Section 4.3). Hence, we believe that indicating LogicQA and ReClor include richer reasoning formats will not necessarily undermine the impacts of finetuning on LogicBench.

Reviewer comment (Evaluation):
Intuitively, PL should be easier than FOL, but the analysis does not bring any insight into why this is not the case with LLM reasoning.

Author response:
Justification of lower performance of GPT-4 on PL as compared to FOL  

A2: Thank you for raising this point. Please note that the results in Table 6 show an average improvement in FOL results because of LLMs’ high accuracy on two axioms, EI and UI. However, when we compare the performance across the six common inference rules between PL and FOL (MT, HS, DS, BD, CD, DD), GPT-4 achieves an average of 49.16% A(Yes) for PL and 39.54% A(Yes) for FOL which shows that our results show the expected behavior.

The high accuracy of GPT-4 in handling EI and UI can be attributed to their simplicity.  While from human experience and complexity theory, FOL is harder than PL in general; in the LLM context, the crucial factor becomes what kind of logical sentences LLMs are pre-trained on. It seems that LLMs are pre-trained more on simple FOL sentences than on simple PL sentences. To get an indication of this we gave GPT-4 prompt_1 - “Give twenty statements that have ‘if’ and ‘then’ in them” and prompt_2 - “Give twenty statements that have material implication "or" in them”. From the results, we can observe that 11 of the 20 sentences in response to the first prompt were FOL kind and only 9 were propositional kind. On the other hand, all 20 statements in the response to the second prompt were FOL sentences. This shows that LLMs’ comprehend simple FOL sentences, thus, showing high accuracy on simpler FOL axioms, i.e., EI and UI. However, LLMs exhibit consistently low performance across other complex/longer inference rules that are common to both PL and FOL (e.g., CD, BD, etc.).

Reviewer comment (Experiments):
The main concern is that the dataset is synthetically generated with pre-defined logical rules and templates. Although this gives a comprehensive categorization of the logic types, it also leaves the question of how well it can generalize to real-world logical reasoning tasks? Although in Table 7 the authors showed that fine-tuning T5 with LogicBench leads to better performance, the performance improvement is quite marginal.

Author response:
Thanks for this comment. Please note that the core contribution of our proposed work is the diagnostic benchmark for systematically evaluating the logical reasoning ability of LLM across 25 different reasoning inference rules. This benchmark goes beyond simple inference rules in classical logic and incorporates more complex inference rules including NM reasoning. Moreover, the evaluation of various LLMs on LogicBench(Eval) leads to interesting findings and shows shortcomings in their ability to accurately solve even single-step reasoning problems. Furthermore, with our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

Reviewer comment (Experiments):
A salient drawback is the synthetic nature of the dataset, crafted using predetermined logical rules and templates. This raises the question of its applicability and relevance to real-world logical reasoning challenges. While the authors showed in Table 7 that fine-tuning T5 with LogicBench leads to better performance, this enhancement was relatively slight.

Author response:
Thanks for this comment. Please note that the core contribution of our proposed work is the diagnostic benchmark for systematically evaluating the logical reasoning ability of LLM across 25 different reasoning inference rules. This benchmark goes beyond simple inference rules in classical logic and incorporates more complex inference rules including NM reasoning. Moreover, the evaluation of various LLMs on LogicBench(Eval) leads to interesting findings and shows shortcomings in their ability to accurately solve even single-step reasoning problems. Furthermore, with our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

Reviewer comment (Evaluation):
The few-shot learning performance from Table 13 hint at the potential of LogicBench to soon become less challenging. Remarkably, with few-shot prompting, GPT-4 already achieves an accuracy of roughly 90% in PL and FOL and about 85% for NM.

Author response:
In the few-shot setting, we provide exemplars corresponding to the inference rule required to answer the test question. Bigger models such as GPT-4 have been shown to be remarkably good at following the in-context exemplars and mimicking the process to reach the correct conclusions. Thus, leveraging the in-context exemplars, GPT-4 achieves high accuracy in a few-shot setting (Table 13). In contrast, the zero-shot setting evaluates the true capabilities of these models in carrying out logical reasoning, as we can not expect that oracle exemplars corresponding to particular logical inference rules will always be available as part of prompts. We also believe that something as fundamental as logical reasoning ability should be an inherent capability of the model which is evaluated in a zero-shot setting. We discuss this point in Section 4.1.

Reviewer comment (Experiments):
The paper lacks the evaluation on modern open-sourced LLMs such as LLAMA, Alpaca, Vicuna, etc.

Author response:
Thanks for this suggestion. We added an evaluation of LLaMa-2-7B on LogicBench(Eval) in Appendix G. Similar to the other LLMs such as GPT-3/4 in zero-shot setting, LLaMa-2 does not fair well on LogicBench(Eval).

Reviewer comment (Experiments):
Since the dataset is quite synthetic, I assume that when fine-tuning the model on a few examples, the model can achieve quite good performance. It would be beneficial for the authors to report this supervised performance as a reference for the potential upper-bound performance.

Author response:
As rightly mentioned by the reviewer, LogicT5 indeed achieves high accuracy (∼97%) on LogicBench(Eval).

Reviewer comment (Evaluation):
Given the first weakness, could the authors elaborate on whether high/low performance on LogicBench truly indicates high/low real-world logical reasoning capabilities for LLMs?

Author response:
While we agree that real-world logical reasoning can be more complex and require multi-step reasoning, we would like to emphasize that our benchmark represents the first step in evaluating LLM across a spectrum of logical inference rules and shows that these LLMs struggle even with the single-step reasoning process when logical rules become complex.  Furthermore, errors made during the initial single-step reasoning can propagate through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios (which is more human-like reasoning). While our benchmark primarily addresses single-step reasoning and doesn't specifically delve into multi-step human-like reasoning, evaluation on LogicBench is a strong indicator of real-world logical reasoning capabilities for LLMs.

Reviewer comment (Experiments):
Given the first weakness, could the authors elaborate on whether there is evidence to support the premise that training LLMs on synthetic logic data such as those in LogicBench genuinely enhances their general logical reasoning capabilities?

Author response:
Some recent works such as [1] show that training on synthetic data can improve the performance of out-of-domain logical reasoning datasets indicating enhanced general logical reasoning capabilities.

[1] https://arxiv.org/pdf/2310.00836.pdf

Reviewer comment (Novelty):
Notwithstanding my #1 strength above, I have to say that I have mixed feelings about the overall utility of this work to reasoning. In a sense, LogicBench is a sort of advanced version of ProofWriter type datasets, and so while I think it will have some marginal utility in the field, it also takes it in the wrong direction (in my opinion) by focusing on a set of clearly delineated but complex and unusual inference steps, whereas advanced "human-like" reasoning for LLMs will require doing multiple steps of reasoning in a very complex and realistic context where there are multiple sources of ambiguity to contend with. Examples derived from templated inference rules don't solve this problem.

Author response:
We note that our benchmark represents a crucial first step in evaluating a variety of LLMs across a spectrum of logical inference rules and shows that these LLMs struggle even with the single-step reasoning process when logical rules become complex.  Furthermore, errors made during the initial single-step reasoning can be propagated through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios (which is more human-like reasoning). While our benchmark primarily addresses single-step reasoning and doesn't specifically delve into multi-step human-like reasoning, it leads to valuable findings that LLMs struggle with even single-step reasoning for complex inference rules. Moreover, even in inference rules (e.g., HS, CD, DD, etc.), multiple sub-steps are involved in matching with the appropriate parts of the inference rule.

Reviewer comment (Evaluation):
My biggest concern is that the results of GPT-4 on even the simpler PL problems don't even beat a majority baseline of 50%, which seems very unexpected and possibly a bug of some kind. The authors do not even mention this surprising finding, which makes me wonder if I have misunderstood the experiments (Each task is binary Y/N, so random should be 50%?).

Author response:
Each task is a binary classification task with a Yes/No answer. Note that, the results in Table 6 show that GPT-4 demonstrates superior performance for simpler PL inference rules such as HS and MT as compared to others in PL which is expected. However, as we introduce more complex PL inference rules (i.e., longer inference rules that are not commonly used in human writing) such as DD, BD, and CD, GPT-4 starts to show a drop in performance. This is one of the findings in the paper as discussed in Section 4.3 (“Longer inference rules are still challenging”).

Reviewer comment (Evaluation):
With a complex benchmark like this, one generally hopes that there have been systematic checks for the integrity and reliability of the benchmark as having sensible answers and tracking the phenomenon to be measured. The authors only offer a vague assurance in the end of sec 3.3 without concrete evidence; I would love to have seen something like a small-scale human eval on a subset to confirm the validity of the benchmark.

Author response:
Thank you for this suggestion. Please note that, in Sec. 3.3, we performed a systematic evaluation to ensure that generated instances follow the intended logical structure. To further support the integrity and reliability of the benchmark, we hired three graduate student volunteers to manually check the quality of generated data instances. Due to the limited rebuttal period, we randomly selected 5 instances from each inference rule, resulting in a total of 125 instances across 25 reasoning patterns for human evaluation. In particular, annotators are asked to provide binary answers (yes/no) for “Validity of generated context”, and “Validity of generated question” to make sure they adhere to the intended logical structure. Each instance is annotated by three different annotators.  The inter-annotator agreement, measured with raw/observed agreement is 0.956. When there was a disagreement between annotators, a majority class was preferred. We provided example task instructions at https://anonymous.4open.science/r/LogicBench-EEBB/human_eval/readme.md. Results reveal that, after majority voting, annotators provided ‘yes’ for all 125 instances showing that all given instances adhere to the intended logical structure which further supports the quality of data checked by authors. We added this study in Appendix L.

Reviewer comment (Writing):
Overall the paper is fairly clear in what it is doing, but there are places where the authors are vague, especially sec 4.3 first section. Most of the points in this section aren't big surprises.

Author response:
Thanks for this comment. Please note that we performed qualitative analysis over selected samples (where the reasoning chain leads to incorrect predictions) to identify flaws in the reasoning process of LLMs, hence, low performance. We believe that different findings across various reasoning types are useful in terms of understanding mistake that is made by models in general. Moreover, this qualitative analysis supports the difficulty posed by negations and the challenges associated with longer inference rules.

Reviewer comment (Novelty):
You say this is the first benchmark for NMR, but there have been others e.g. BoardGameQA [1].

Author response:
Thank you for this suggestion. BoardgameQA focuses on reasoning with incomplete and contradictory information, whereas, our benchmark systematically categorizes and studies different patterns for NM reasoning. We will include this in the related work in our revised version.

Reviewer comment (Presentation):
Table 2, I think the definition of EI is backwards.

Author response:
Thanks for highlighting this point again. This is indeed an oversight from our side. The definition of EI is indeed backward as you pointed out. What we meant was EG (​​Existential generalization - https://en.wikipedia.org/wiki/Existential_generalization). We have corrected this in the revised version.

Reviewer comment (Evaluation):
It seems when we generate natural language instantiations of each rule, we expect 'if' to always be treated as material implication, but it is well known that depending on the context 'if' can be interpreted in many other ways e.g. as a subjunctive, so it may be the case that the more natural interpretation of a particular context sentence generated by the pipeline is such. Is this a problem for the reliability of the benchmark?

Author response:
Thanks for bringing up this interesting point. While the use of ‘if’ in natural language sentences in the wild could have other interpretations such as a subjunctive, we took care in our construction so that in our dataset ‘if’ meant material implication. Specifically, our data creation pipeline ensures the logical correctness of generated context and questions where ‘if’ is used as material implication, thus maintaining the reliability of our instances.

Reviewer comment (Reproducibility):
Is LogicT5 pre-trained from scratch on LogicBench(Aug) using the T5 model recipe or do you train on LogicBench starting from a pretrained T5-Large checkpoint?

Author response:
We train T5 on LogicBench(Aug) starting from a pre-trained T5-Large checkpoint.

Reviewer comment (Experiments):
Sec 3.2.1: the purpose of this step is to generate single atomic propositions. It seems we do this by generating entire triples for particular inference steps (like Modus Tollens) and then only extracting the individual atomic propositions. Why this complicated process?

Author response:
We would like to clarify that during this step, we only generated single atomic propositions rather than the complete triplet. In the prompt, we instruct the model with entire triplets as exemplars, aiming to provide an understanding of how the generated sentences will be applied in later stages of data creation. The objective of presenting such examples is to encourage the model to produce more coherent sentences, consequently enhancing the naturalness of generated instances. While our formatting instructions explicitly indicate the generation of only atomic propositions, we provide a complete prompt in Appendix A (Figure 4) for further clarification.