Paper: On the Paradox of Generalizable Logical Reasoning in Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
An experiment should try to isolate the hypothesis being tested and remove confounding factors. I am also suspicious of your conclusions; e.g., you claim "In other words, LLMs show significantly worse performance when semantics are decoupled" but without error bars, the Symbols and Semantic of Table 2 look the same to me. I don't think that you should generalize from a single example (Symbol tree), where the results don't hold for another example (ProofWriter).

Author response:
We have added error bars to the ChatGPT results in the revised version. Due to the cost of GPT-4, we do not provide its error bars. As seen in Table 1, the semantic performance under error bars is still significantly worse than symbolic performance. 

As for Table 2, we observed that the symbol version performs comparably to the semantic version, primarily because the semantics in ProofWriter are irrelevant to commonsense. Essentially, tasks which require less intricate semantic understanding or minimal dependence on common sense knowledge are not significantly impacted by the presence or absence of semantics in the language models' performance, as highlighted in blue.

To further validate our finding that Language Learning Models serve as in-context semantic reasoners rather than symbolic reasoners, we introduced a counter-commonsense setting and conducted more ablation studies, as shown in Table 3. In addition to deduction tasks, we executed comprehensive experiments in induction and abduction, to provide a holistic illustration of our conclusions.

Reviewer comment (Evaluation):
Much deductive reasoning is combinatorially difficult, and is difficult even for humans. I'm surprised humans can do the examples B.2 well. I thought that the psychology literature results are that humans are not good at logical reasoning -- but I am not a psychologist.

Author response:
Piaget's theory of formal operations (https://en.wikipedia.org/wiki/Piaget' s_theory_of_cognitive_development) points out that individuals in early to middle adolescence are capable of hypothetical and deductive reasoning that involves assumptions with no necessary relation to reality, also including counterfactual thinking. While not all individuals use formal reasoning in their daily lives [1], most humans possess this skill to some degree. And they possess the cognitive ability to engage in formal reasoning when required. In addition to Piaget's theories, it has been suggested by citations [1][2][3] that humans possess symbolic reasoning abilities. To rigorously verify this claim, we will conduct a human study to measure human performance in these tasks. 

[1] https://cbmm.mit.edu/research/2013-2018-research/exploring-future-directions/learning-and-reasoning-symbolic-domains
[2] https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00275/full
[3] https://tigerprints.clemson.edu/all_dissertations/2932/

Reviewer comment (Theory):
You have a strange definition of abduction. It looks like "find a proof" (from the only example given in Figure 1 and on page 4), where the sisterOf and motherOf are given as facts. Abduction in logic means to find a plausible explanation: e.g., why did someone cough? An explanation is that they have asthma. Another explanation is they have a cold. The system does not know whether they have asthma or a cold. It is difficult to judge the correctness of an answer.

Author response:
Abduction in logic traditionally involves generating plausible explanations for a given observation. Indeed, our approach to abductive reasoning aligns with this definition, particularly within the framework of first-order logic. Based on the definition of Logic-based abduction (https://en.wikipedia.org/wiki/Abductive_reasoning), a proof-theoretical abduction method for first-order logic have been proposed for finding explanations. In first-order logic contexts, abduction typically involves reasoning that is more definitive and less speculative than in commonsense reasoning scenarios. Most abductive reasoning in this domain tends to yield plausible conclusions.

Moreover, Regarding the specific task settings, our methodology follow the approach "Explanation Generation with Theory" described in the paper https://arxiv.org/pdf/2303.12023.pdf. This approach involves generating a new hypothetical fact h such that, when combined with a given theory C, can prove an observation O, i.e., $C \cup {h} |=O$. We also reference "Explanation Classification," where the task is to choose the most plausible hypothesis from given alternatives to explain sequential observations. Considering the complexity of generating new facts and the necessity of a feasible evaluation framework, we adapted our task to involve selecting the most plausible facts from existing ones. This adaptation allows us to maintain the rigors of first-order logic abduction while ensuring that our tasks are feasible and effectively evaluative.

Reviewer comment (Experiments):
Why only "zero-shot Symbols" for humans? Who are the humans used? Are they trained in logic? (Appendix F1 doesn't provide much detail beyond being diverse college/graduate students). This is not up to the standards of a good human experiment to make any conclusions. Were the humans all the same? Why isn't there a range? The examples you gave on p. 36 for humans were for the semantics case (unless I misunderstood appendix I). I wish your appendices gave a few complete examples rather than more abstract examples; what was the actual input and what was the actual output for the computers and the humans?

Author response:
We apologize for any confusion regarding Appendix I, which was designed to examine the influence of irrelevant information on logical reasoning. We will provide more details about the human study in the revised paper. 

We recruited 11 participants from diverse science and engineering backgrounds, including computer science, electronics, artificial intelligence, and automation. Although they have basic understanding of simple logic concepts, they are not experts in logical reasoning. Therefore, we provided them with task instructions that explained the concepts of deduction, induction, and abduction, aligned with the illustrations and definitions of logical reasoning presented in Section 3 of our paper.

We then presented them with 18 specific tasks, including six tasks for each deductive, inductive, and abductive reasoning type. Each task closely resembled the zero-shot prompts given to LLMs. We refer to this setting as "zero-shot" because we did not provide any further specific examples to help participants understand the tasks. The examples can be found in Appendix I and detailed materials have been uploaded in supplementary.

Reviewer comment (Experiments):
For the Symbolic Tree dataset, is the LLM/human told it is a closed-world dataset? What is a false fact? None of the examples seem to rely on negative facts, and none of the examples in appendix B have any. Are there negative examples as well as positive examples?

Author response:
In our experimental setting, Symbolic Tree is a closed-world dataset, meaning that any facts not presented or inferable in the dataset are assumed to be false. In other words, when the language model (LLM) outputs an "Unknown" answer, we interpret it as "False". 

Consider the following false case example:
```
Statement: $r27$(patrick, marko)
Answer: We can use logical rule L24, which states that if there is a relationship of type 3 between B and A, and a relationship of type 3 between C and B, and A has property 1, then A has relationship 27 with C. 
From the facts, we know that there is a relationship of type 3 between Patrick and Theodor (F43), and a relationship of type 3 between Theodor and Marko (F44). However, we do not have any direct information about a relationship of type 1 involving Patrick. Therefore, we cannot use L24 to prove $r27$(patrick, marko). 
Thus, we cannot determine whether the statement $r27$(patrick, marko) is True or False using the given logical rules and facts.
```

Reviewer comment (Evaluation):
How hard are the induction tasks? What bias do you assume for the correct answer? Why should we believe the "gold proofs" are correct?

Author response:
In our induction tasks, we utilized the Symbolic Tree dataset, which consists of approximately 99 basic facts including gender information ("r43" and "r44"), "parentOf" ("r1"), and "inverse_parentOf" ("r45") relationships. Additionally, there are about 5-15 inferred facts related to the unknown (to be predicted) rule.

Note that a single rule can be equivalent to multiple rules. For example, the rule $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$ can be represented as two separate rules: $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \rightarrow \text{GrandparentOf}(x,z)$ and $\forall x,y,z: \text{GrandparentOf}(x,z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$. To simplify the induction evaluation process, we rewrite all logical rules by including only the "parentOf" and "gender" relations in the rule body. Doing so ensures that each inferred relation is implied by a single logical rule. We provide a rule template to define the rule search space, as detailed in Appendix A.2, and specific examples can be found in Appendix E.3.

Reviewer comment (Evaluation):
Can you explain why in Table 1, for ChatGPT Zero-Shot-CoT is better than Zero-Shot for deduction, but in Table 2, it is worse for all depths? Does CoT help?

Author response:
Indeed, in Table 1, we observe that the performance of ChatGPT in the Zero-Shot-CoT setting does not show a marked improvement over the Zero-Shot setting, especially when considering error bars. This suggests that the addition of CoT does not significantly enhance the model's ability to solve deduction tasks.

Table 2 presents an even more pronounced discrepancy, where CoT settings perform worse than the Zero-Shot setting. We hypothesize two main reasons for this observed trend:

1. The CoT settings perform worse than the zero-shot setting with a higher frequency of "Cannot be determined" answers. We speculate that this may arise from the LLMs struggling to generate explicit reasoning traces effectively.
2. Utilizing a step-by-step explicit reasoning trace might lead to error propagation, resulting in worse performance than directly obtaining the answer. In addition, this effect seems to be more pronounced in tasks with semantic contexts, as seen in Table 2. The discrepancy between Zero-Shot-CoT and CoT is greater. This observation might indicate that step-by-step reasoning could amplify the distracting effect of weird semantics, such as "The squirrel needs the dog."

Consequently, CoT might not always be helpful for reasoning tasks that involve in-context new knowledge for models with weaker reasoning abilities like ChatGPT.

Reviewer comment (Writing):
For "Paradox 1" and "Paradox 2" - why are they paradoxes? Maybe use "hypothesis"?

Author response:
We would like to explore and highlight two key paradoxical observations in the capabilities of Large Language Models (LLMs) with respect to logical reasoning.

**Paradox 1** revolves around the unexpected discrepancy in LLMs' performance in logical reasoning tasks. While LLMs like GPT-4 show high accuracy (91.1%) in tasks rich in semantics, their performance significantly declines when semantic cues are replaced with symbolic representations. This observation is paradoxical because, according to the formal definition of logical reasoning, LLMs' reasoning ability should rely purely on the form or syntax of premises and conclusions, independent of semantic content. Yet, our findings indicate that LLMs heavily rely on semantics, suggesting that current logical reasoning tasks may not fully reflect the true extent of LLMs' formal reasoning capabilities.

**Paradox 2** emerges from our fine-tuning experiments on symbolic reasoning tasks. While supervised fine-tuning narrows the performance gap between semantic and symbolic settings, enabling LLMs to perform logical reasoning seemingly agnostic to semantics, their ability to generalize to unseen logical rules remains limited. This indicates that LLMs primarily use template matching mechanisms for reasoning, rather than engaging in genuine formal reasoning processes. This paradox challenges the common perception that fine-tuning effectively enhances LLMs' logical reasoning skills, suggesting that the improvement may be superficial rather than indicative of true reasoning abilities.

In summary, our paper identifies two paradoxes: 1) **LLMs' reliance on semantics rather than engaging in formal reasoning, despite seeming logical reasoning proficiency**, and 2) **the superficial nature of improvements in logical reasoning through fine-tuning, which fails to achieve deep generalization to new rules**. These paradoxes question the true logical reasoning capabilities of LLMs and emphasize the need for more nuanced understanding and development in this area.

Reviewer comment (Evaluation):
The first part of the claims are made by direct prompt ChatGPT/GPT4. However, some gaps between the performances are not significant.

Author response:
In Table 1, considering error bars, semantic performance is still notably worse than symbolic performance.

As for Table 2, we observed that the symbol version performs comparably to the semantic version, primarily because the semantics in ProofWriter are irrelevant to commonsense. Essentially, tasks which require less intricate semantic understanding or minimal dependence on common sense knowledge are not significantly impacted by the presence or absence of semantics in the language models' performance, as highlighted in blue.

To further validate our finding that Language Learning Models serve as in-context semantic reasoners rather than symbolic reasoners, we introduced a counter-commonsense setting and conducted more ablation studies, as shown in Table 3. In addition to deduction tasks, we executed comprehensive experiments in induction and abduction, to provide a holistic illustration of our conclusions.

Reviewer comment (Theory):
Some claims are too general to be valid, please check the question part.

Author response:
Re W2 (1): The results of symbolic reasoning evaluation are provided in Appendix J.2. In this section, we perform deductive and inductive reasoning experiments while providing the models with only relevant facts. The findings demonstrate that LLMs are easily disrupted by irrelevant information, especially when confronted with longer contexts.

Re W2 (2): We suppose the reviewer is concerned about "rep_all_with_counter" setting presented in Table 4. We would like to emphasize that even though fine-tuned LLMs can leverage template matching to make predictions, it is still inevitable that they may become distracted by prior knowledge.

Re W2 (3): Our paper focuses on first-order logical reasoning abilities. Unlike the inference of machine learning which usually generalizes to the same distribution, an agent with “true” logical reasoning abilities can reason with any given information to solve new, unfamiliar problems, regardless of any prior knowledge or different knowledge representations. In other words, it can generalize across novel rules (Sec 1 Line 13). For instance, consider training on $r1(A, B) \land r2(B) \rightarrow r3(B, A)$ and testing on $r1(A, B) \land r2(B, C) \rightarrow r3(C, B)$, as depicted in Fig 3. In our experimental setting, RuDaS is the dataset featuring rules with similar forms as $\text{xx} \land \text{xx} \cdots \rightarrow \text{xx}$. FOLIO is a complex and diverse dataset for logical reasoning expressed in natural language. Through our comprehensive experiment results, we find fine-tuned LLMs on symbolic logic datasets cannot generalize to novel rules, highlighting that fine-tuning LLMs fails to achieve generalizable logic reasoning.

Reviewer comment (Theory):
For the claim regarding learning to reason, why do authors expect that "fine-tuning the LLM on symbolic trees" should lead to good performance in FOLIO and RuDaS?

Author response:
Re W2 (3): Our paper focuses on first-order logical reasoning abilities. Unlike the inference of machine learning which usually generalizes to the same distribution, an agent with “true” logical reasoning abilities can reason with any given information to solve new, unfamiliar problems, regardless of any prior knowledge or different knowledge representations. In other words, it can generalize across novel rules (Sec 1 Line 13). For instance, consider training on $r1(A, B) \land r2(B) \rightarrow r3(B, A)$ and testing on $r1(A, B) \land r2(B, C) \rightarrow r3(C, B)$, as depicted in Fig 3. In our experimental setting, RuDaS is the dataset featuring rules with similar forms as $\text{xx} \land \text{xx} \cdots \rightarrow \text{xx}$. FOLIO is a complex and diverse dataset for logical reasoning expressed in natural language. Through our comprehensive experiment results, we find fine-tuned LLMs on symbolic logic datasets cannot generalize to novel rules, highlighting that fine-tuning LLMs fails to achieve generalizable logic reasoning.

Reviewer comment (Evaluation):
Authors can also perform additional interpretability analyses to help the community understand the failure modes.

Author response:
We provided some failure cases and discussions in Appendix E. These failures can likely be attributed to interference from irrelevant information and the inclusion of hallucinated information.

Reviewer comment (Reproducibility):
Can authors clarify which version of GPT4 and ChatGPT they use? There are many timestamped versions with differing context length.

Author response:
We used the gpt-3.5-turbo-0301 and gpt-4-0314 versions for this study. We will provide this detail in the revised paper.

Reviewer comment (Novelty):
The overall pitch of the paper is not as convincing as it could be. It's written like the community believes (from prior papers) that LLMs have strong logical reasoning skills, and that the current paper questions this belief and provides evidence against it. However, I don't think it's the case that the community believes logical reasoning is solved by LLMs. E.g., for the datasets considered here, even the baseline performance (in the original, so-called *Semantics* version of the tasks) is not high enough. This makes the motivation of the study relatively weak.

Author response:
Indeed, Large Language Models have achieved impressive performance on a variety of reasoning tasks[1][2], such as arithmetic and commonsense reasoning. However, the extent to which LLMs have mastered formal logical reasoning remains unclear. Previous studies have shed light on LLMs’ reasoning abilities to some extent, but a deeper exploration of their underlying reasoning mechanisms is still necessary. Our work aims to fill this gap by providing a novel perspective, going beyond mere performance metrics to understand the essence of LLMs' reasoning processes.

The performance on the Symbolic Trees semantics version has achieved commendable results (91% accuracy on GPT-4). However, performance on reasoning datasets with semantics does not necessarily reflect a deep understanding or mastery of formal reasoning. When the semantics are decoupled, LLMs tend to perform much worse. Thus, our primary goal is to investigate whether LLMs possess genuine formal reasoning abilities or simply exploit superficial semantic associations between questions and answers to make intuitive predictions. This investigation is of great importance for advancing the development of artificial general intelligence (AGI).

Furthermore, we also experiment with fine-tuning Llama-2 on pure symbolic reasoning tasks to bridge the gap. Although it appears to close the gap, this is achieved through template matching rather than invoking the fundamental formal reasoning abilities. We identify this phenomenon to provide a clear understanding of the boundaries within which LLMs currently solve logical reasoning tasks.

Therefore, the interesting findings aim to motivate further investigation into unveiling the inner workings of black-box LLMs.

[1] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.
[2] OpenAI. Gpt-4 technical report, 2023. 1

Reviewer comment (Writing):
The pitch is also confusing because of the use of the word "paradox". What paradox exactly is being explored here? Reading the title, I was expecting to see something like: LLMs are great at X, which implies they should be great at Y too, but they fail at Y, raising a conundrum. Or some such internal conflict or conflict with commonsense expectations, that would justify the word paradox. I'm not sure what the authors have in mind for a paradox.

Author response:
We would like to explore and highlight two key paradoxical observations in the capabilities of Large Language Models (LLMs) with respect to logical reasoning.

**Paradox 1** revolves around the unexpected discrepancy in LLMs' performance in logical reasoning tasks. While LLMs like GPT-4 show high accuracy (91.1%) in tasks rich in semantics, their performance significantly declines when semantic cues are replaced with symbolic representations. This observation is paradoxical because, according to the formal definition of logical reasoning, LLMs' reasoning ability should rely purely on the form or syntax of premises and conclusions, independent of semantic content. Yet, our findings indicate that LLMs heavily rely on semantics, suggesting that current logical reasoning tasks may not fully reflect the true extent of LLMs' formal reasoning capabilities.

**Paradox 2** emerges from our fine-tuning experiments on symbolic reasoning tasks. While supervised fine-tuning narrows the performance gap between semantic and symbolic settings, enabling LLMs to perform logical reasoning seemingly agnostic to semantics, their ability to generalize to unseen logical rules remains limited. This indicates that LLMs primarily use template matching mechanisms for reasoning, rather than engaging in genuine formal reasoning processes. This paradox challenges the common perception that fine-tuning effectively enhances LLMs' logical reasoning skills, suggesting that the improvement may be superficial rather than indicative of true reasoning abilities.

In summary, our paper identifies two paradoxes: 1) **LLMs' reliance on semantics rather than engaging in formal reasoning, despite seeming logical reasoning proficiency**, and 2) **the superficial nature of improvements in logical reasoning through fine-tuning, which fails to achieve deep generalization to new rules**. These paradoxes question the true logical reasoning capabilities of LLMs and emphasize the need for more nuanced understanding and development in this area.

Reviewer comment (Novelty):
Overall, while I thought the study was useful, I didn't find anything subjectively surprising. It is generally accepted that LLMs --- being **language models** --- rely on many linguistic clues and prior knowledge to perform tasks. Taking away these clues is thus expected to drop performance. Similarly, training directly on the so-called *Symbolic* form should help, which the authors also find to be the case. All this is very much aligned with expectation, which makes it difficult to pin point what the new knowledge this paper would bring to the community.

Author response:
Please refer to response to W1 and W2. Here, we would like to highlight our study's novelty and contributions:


We aim to investigate whether LLMs have strict formal reasoning abilities or simply exploit the superficial semantic associations between questions and answers to make intuitive predictions as well as using template matching of the known knowlege, which is crucially important to AGI development. To answer the research question, we decouple semantics from the language reasoning process and force LLMs to perform symbolic reasoning to finish three kinds of reasoning tasks (deduction/induction/abduction) as well as inverstigating the effect of fine-tuning LMs.

* We find that when provided with symbolic knowledge in the form of facts and rules, LLMs struggle to perform simple deduction reasoning tasks. These tasks, which are relatively easy for the average human, only require following deductive rules and making logical inferences, as the human performance states shown in Table 1.
* We also compare the reasoning abilities of LLMs across induction and abduction tasks and find that they perform notably worse in these types of reasoning tasks compared to deduction, regardless of whether semantics or symbols are used. When semantics are decoupled, the drop in performance is even more significant. These findings highlight the considerable room for improvement in LLMs' reasoning abilities and suggest that relying solely on semantics to achieve symbolic reasoning is challenging.
* When fine-tuning on symbolic reasoning data, supervised fine-tuning considerably narrows the performance gap between symbolic and semantic settings, enabling LLMs to perform logical reasoning independent of semantics. However, when generalized to unseen logical rules, the fine-tuned model's performance significantly drops. This indicates that LLMs rely on a template-matching mechanism rather than a genuine formal reasoning process.
* In addition to the aforementioned findings, we have conducted numerous fine-grained experiments. These meticulous experimental results provide valuable insights and further strengthen our work.

Reviewer comment (Presentation):
There are number of additional side experiments in the paper. This, in principle, is nice. However, while reading through those section, I found the large number of questions to be somewhat distracting. At the least, the authors should try to thread the narrative better through these side experiments and analyses, and try to provide a view of them that helps support the overall message of the paper.

Author response:
Thank you for your suggestions. While we designed additional side experiments and performed in-depth analyses in order to provide more insights, we understand that they can seem distracting. In the revision, we will re-organize the paper and strive to strengthen the connection between the side experiments and the overall message of the paper, ensuring a more cohesive narrative throughout the manuscript.

Reviewer comment (Novelty):
In summary, while it's somewhat useful to see the experiments on *Symbolic* forms of the considered datasets done, the results don't really feel different from what one might expect to see.

Author response:
Re W5 (1): Thank you for pointing out the potential confusion related to the use of the term "semantics". In our paper, we indeed refer to linguistic semantics. We appreciate your feedback and will ensure to clarify and emphasize the linguistic aspect of semantics in our revised manuscript.

Re W5 (2): We will also cut down on the number of citations and focus on providing focused explanations of the relationships and differences between our work and the selected prior works.

Re W5 (3): We revised and restructure the sentence for better clarity and readability in the updated version of the paper.

Reviewer comment (Writing):
The use of *Semantics* when referring to relation names but not when referring to logic is confusing. Logic, of course, by design has a very clear and unambiguous semantics. I think what you mean is *linguistic semantics* of predicate names. If so, please be sure to clarify this and emphasize the *linguistic* aspect.

Author response:
Re W5 (1): Thank you for pointing out the potential confusion related to the use of the term "semantics". In our paper, we indeed refer to linguistic semantics. We appreciate your feedback and will ensure to clarify and emphasize the linguistic aspect of semantics in our revised manuscript.

Reviewer comment (Writing):
Your related work section (as well as the introduction) has a **lot** of citations, almost too many to be meaningfully valuable. E.g., near the top of page 3, you have 12+ citations around ICL, without any explanation of the connection between these prior works and what's in your paper. As a general rule, it's more valuable for the related work section to point out the few most related works AND articulate a clear connection of the current work to them, as opposed to dumping a huge list of papers just to cover every possible connection.

Author response:
Re W5 (2): We will also cut down on the number of citations and focus on providing focused explanations of the relationships and differences between our work and the selected prior works.

Reviewer comment (Writing):
The last sentence of page 2 ("Wei et al propose symbolic tuning, which ....") is very long and hard to parse.

Author response:
Re W5 (3): We revised and restructure the sentence for better clarity and readability in the updated version of the paper.

Reviewer comment (Writing):
What exactly is the paradox (i.e., some form of commonsense contradiction) that you are referring to? Or if *paradox* is not the right word, please replace it with something else.

Author response:
Please refer to our response to W2.

Reviewer comment (Evaluation):
Looks like you forgot to discuss Table 2 (results on ProofWriter) in the main paper. What are the main take-aways from this table and how do they support your claims? E.g., it appears that going from the Semantics setting to the Symbolic setting does *not* reduce the performance of the models substantially; in fact, the performance goes up in many cases. How does this align with your claims from Table 1?

Author response:
Our discussion regarding Table 2 can be found on Page 7, Paragraph 1 under the subheading "When are semantics irrelevant to prior knowledge?". To reiterate, we observed that the symbol version performs comparably to the semantic version because the semantics in ProofWriter are irrelevant to commonsense. Essentially, tasks which require less intricate semantic understanding or minimal dependence on common sense knowledge are not significantly impacted by the presence or absence of semantics in the language models' performance.

Reviewer comment (Novelty):
The main result for prompting was that models performed significantly worse when symbols were used to describe the rules instead of meaningful words. This question has been studied before in prior work cited by the authors (PrOntoQA and Dasgupta et al.), where made-up words were used instead of short symbols like A, B, etc., but the insight seems the same. If the authors believe their finding is novel, this did not come across in the paper.

Author response:
> W1. similar question about content effect in LLMs' reasoning has been studied before.

While they conclude that "Language models show human-like content effects on reasoning", this is only a subordinate conclusion that suggests LLMs are in-context semantic reasoners influenced by semantic associations. However, their conclusion does not directly imply that "LLMs are not symbolic reasoners," as suggested by Paradox 1: "LLMs are in-context semantic reasoners rather than symbolic reasoners".


According to the definition of deductive reasoning from Wikipedia (https://en.wikipedia.org/wiki/Deductive_reasoning), logical reasoning should be formal and depends only on the form or the syntax of the premises and the conclusion, irrespective of the specific contents of this argument. To this end, our task decouples semantics and focuses on pure symbolic reasoning (only provided syntax). When we feed these to LLMs, they show significantly worse performance compared to when normal semantic words are fed. This phenomenon indicates that LLMs fail to invoke the basic formal reasoning abilities of humans but instead rely on shallow semantic associations for prediction. 

Moreover, we conduct comprehensive experiments on various logical reasoning, including deduction, induction and abduction tasks and find that they perform notably worse in these types of reasoning tasks compared to deduction, regardless of whether semantics or symbols are used. When semantics are decoupled, the drop in performance is even more significant. These findings highlight the considerable room for improvement in LLMs' reasoning abilities and suggest that relying solely on semantics to achieve symbolic reasoning is challenging.

Reviewer comment (Reproducibility):
The performance gap with prompting results is significantly larger in the induction and abduction tasks, with GPT-4 scoring under 10% in induction when symbols were used. It is unclear whether this is due to the task setup or genuine model failures. The paper provides little detail about these tasks beyond a general description, and more examples of problems and GPT-4 responses would help verify the setup and performance interpretation. Although the reviewer looked in the Appendix, they could not find these examples.

Author response:
> W2. specific examples of GPT-4 failures in induction and abduction

We provided some failure cases and discussions in Appendix E. These failures can likely be attributed to interference from irrelevant information and the inclusion of hallucinated information.

Reviewer comment (Evaluation):
Can you point to specific examples of GPT-4 failures in induction and abduction?

Author response:
> W2. specific examples of GPT-4 failures in induction and abduction

We provided some failure cases and discussions in Appendix E. These failures can likely be attributed to interference from irrelevant information and the inclusion of hallucinated information.

Reviewer comment (Evaluation):
Generally, the few-shot numbers reported seem worse than zero-shot results. Why would that be the case? This might point to suboptimal examples of reasoning.

Author response:
> Q1: Generally, your few-shot numbers seem worse than zero-shot. Why would that be the case? That might point to not giving good examples of reasoning. Do you also give examples of "false" cases?

In our experimental setting, Symbolic Tree is a closed-world dataset, meaning that any facts not presented or inferable in the dataset are assumed to be false. In other words, when the language model (LLM) outputs an "Unknown" answer, we interpret it as "False". Moreover, we selected our final few-shot demonstrations where LLMs perform better to be those generated by LLMs prompted with "Let's think step by step," which include examples of both true and false cases.

Consider the following false case example:
```
Statement: $r27$(patrick, marko)
Answer: We can use logical rule L24, which states that if there is a relationship of type 3 between B and A, and a relationship of type 3 between C and B, and A has property 1, then A has relationship 27 with C. 
From the facts, we know that there is a relationship of type 3 between Patrick and Theodor (F43), and a relationship of type 3 between Theodor and Marko (F44). However, we do not have any direct information about a relationship of type 1 involving Patrick. Therefore, we cannot use L24 to prove $r27$(patrick, marko). 
Thus, we cannot determine whether the statement $r27$(patrick, marko) is True or False using the given logical rules and facts.
```
Indeed, considering error bars, the results of Few-Shot-CoT are comparable to Zero-Shot-CoT. This finding suggests that the zero-shot capabilities of current large language models are approaching their few-shot learning abilities. A plausible explanation for this development is the enhanced human-alignment and instruction tuning present in current LLMs. These improvements enable the models to better understand the meaning and context of tasks in advance, allowing them to perform well even in zero-shot settings.

Reviewer comment (Theory):
In particular, some examples in the Appendix appear to contain invalid reasoning. For example, the one in Appendix H with Statement: r8(Elena, Nina) and Rule L5 might not be correct, as L5 only proves r8(A, C) is true if premises are satisfied and says nothing about situations where they are not. This example seems misleading and the reasoning template does not generalize. All examples below this one conclude "true." Are "false" cases also provided?

Author response:
> Q1: Generally, your few-shot numbers seem worse than zero-shot. Why would that be the case? That might point to not giving good examples of reasoning. Do you also give examples of "false" cases?

In our experimental setting, Symbolic Tree is a closed-world dataset, meaning that any facts not presented or inferable in the dataset are assumed to be false. In other words, when the language model (LLM) outputs an "Unknown" answer, we interpret it as "False". Moreover, we selected our final few-shot demonstrations where LLMs perform better to be those generated by LLMs prompted with "Let's think step by step," which include examples of both true and false cases.

Consider the following false case example:
```
Statement: $r27$(patrick, marko)
Answer: We can use logical rule L24, which states that if there is a relationship of type 3 between B and A, and a relationship of type 3 between C and B, and A has property 1, then A has relationship 27 with C. 
From the facts, we know that there is a relationship of type 3 between Patrick and Theodor (F43), and a relationship of type 3 between Theodor and Marko (F44). However, we do not have any direct information about a relationship of type 1 involving Patrick. Therefore, we cannot use L24 to prove $r27$(patrick, marko). 
Thus, we cannot determine whether the statement $r27$(patrick, marko) is True or False using the given logical rules and facts.
```
Indeed, considering error bars, the results of Few-Shot-CoT are comparable to Zero-Shot-CoT. This finding suggests that the zero-shot capabilities of current large language models are approaching their few-shot learning abilities. A plausible explanation for this development is the enhanced human-alignment and instruction tuning present in current LLMs. These improvements enable the models to better understand the meaning and context of tasks in advance, allowing them to perform well even in zero-shot settings.

Reviewer comment (Presentation):
Why are there missing entries for induction in Table 1?

Author response:
> Q2: Why are there missing entries in induction in Table 1?

This is because we did not perform a few-shot setting. Symbolic Tree dataset has a limited number of rules, and these rules exhibit a degree of interconnectedness. For instance, a rule like 'grandAuntOf' can be derived through the composition of rules like 'sisterOf' and 'grandparentOf'. The potential overlap and dependencies among the rules could skew the assessment of the model's true inductive reasoning capabilities. Consequently, we chose to only conduct a zero-shot setting where we provide facts to infer new rules without exposing the model to similar examples.

Reviewer comment (Novelty):
What do you think are the novel insights in your experiments with words versus symbols compared to results in prior works on content effects in LLM reasoning?

Author response:
> Q3: What do you think are the novel insights in your experiments with words <--> symbols compared to results in prior works around content effects in LLM reasoning?

Please refer to the reponse to W1 -- similar question about content effect in LLMs' reasoning has been studied before.

Reviewer comment (Experiments):
For induction and abduction, what was the complexity of the held-out premises or rules? How did you ensure the answer was unique, given that in first-order logic an infinite set of hypotheses could derive any given conclusion?

Author response:
> Q4: For induction and abduction, what was the complexity of the held-out premises or rules? How did you make sure the answer was unique, since this is logically non-trivial?


In our induction tasks, we utilized the Symbolic Tree dataset, which consists of approximately 99 basic facts including gender information ("r43" and "r44"), "parentOf" ("r1"), and "inverse_parentOf" ("r45") relationships. Additionally, there are about 5-15 inferred facts related to the unknown (to be predicted) rule.

Note that a single rule can be equivalent to multiple rules. For example, the rule $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$ can be represented as two separate rules: $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \rightarrow \text{GrandparentOf}(x,z)$ and $\forall x,y,z: \text{GrandparentOf}(x,z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$. To simplify the induction evaluation process, we rewrite all logical rules by including only the "parentOf" and "gender" relations in the rule body. Doing so ensures that each inferred relation is implied by a single logical rule. We provide a rule template to define the rule search space, as detailed in Appendix A.2, and specific examples can be found in Appendix E.3.

For abduction tasks, we created the prompts containing gender information and "parentOf" relationships, alongside logical rules consisting of "male", "female", and "parentOf". The dataset contains about 28 logical rules and 63 facts. We evaluate the abilities by generating explanations. Since there can be multiple valid explanations in abduction, as long as an LLMs output one possible interpretation, the reasoning is considered correct. Specific example are provided in Appendix E.1 and E.2.

Further details about task descriptions are included in Appendix G.

Reviewer comment (Reproducibility):
For fine-tuning, can you provide specific fine-tuning examples, beyond just the prompts?

Author response:
> Q5: specific fine-tuning examples

We have included specific fine-tuning examples in Appendix D of our paper.

Reviewer comment (Novelty):
I think the major weakness is the lack of novelty. Previous works [e.g. Saparov & He] already showed that semantics affects LLMs's reasoning ability, and that if we give new fact and rules contrary to the pretraining prior, the model struggles with learning those new rules. I think the main message of this paper is the same thing and not very new.

Author response:
We would like to address your concerns regarding the novelty of our work and stress how our findings on paradox 1 differ from the conclusions drawn in prior works. We will discuss paradox 2 subsequently.

While Saparov and He (2022) conclude that "Language models show human-like content effects on reasoning", this is only a subordinate conclusion that suggests LLMs are in-context semantic reasoners influenced by semantic associations. However, their conclusion does not directly imply that "LLMs are not symbolic reasoners," as suggested by Paradox 1: "LLMs are in-context semantic reasoners rather than symbolic reasoners".


According to the definition of deductive reasoning from Wikipedia (https://en.wikipedia.org/wiki/Deductive_reasoning), **logical reasoning should be formal and depends only on the form or the syntax of the premises and the conclusion, irrespective of the specific contents of this argument.** To this end, our task decouples semantics and focuses on pure symbolic reasoning (only provided syntax). When we feed these to LLMs, they show significantly worse performance compared to when normal semantic words are fed. This phenomenon indicates that LLMs fail to invoke the basic formal reasoning abilities of humans but instead rely on shallow semantic associations for prediction. 

Moreover, we conduct comprehensive experiments on various logical reasoning, including deduction, induction and abduction tasks and find that they perform notably worse in these types of reasoning tasks compared to deduction, regardless of whether semantics or symbols are used. When semantics are decoupled, the drop in performance is even more significant. These findings highlight the considerable room for improvement in LLMs' reasoning abilities and suggest that relying solely on semantics to achieve symbolic reasoning is challenging.

[1]Saparov, Abulhair, and He He. "Language models are greedy reasoners: A systematic formal analysis of chain-of-thought." *arXiv preprint arXiv:2210.01240* (2022).

Reviewer comment (Experiments):
While I agree that looking at test performance on multiple OOD datasets is important, I hope the authors can explain more clearly whether the datasets contain the same logic rules as the training dataset (LogicTree). If they're different, why do we expect finetuning on LogicTree would generalize at all? Requiring the model to generalize to any novel symbolic rule OOD doesn't seem reasonable to me. Usually for domain generalization one has to specify the boundary of domains. Is this all first-order logic or propositional logic? The delineation seems unclear to me, and I'm not sure inductive reasoning is comparable to deductive reasoning, since we would also not want the model to learn spurious correlations in context. I think the authors should clarify the exact scope of expected generalization in mathematical language. For example, we may want to train on 2-hop but generalize to multi-hop problems, etc.

Author response:
We would like to clarify that our paper focuses on first-order logical reasoning abilities, which will be emphasized in the revised version. Unlike the inference of machine learning which usually generalizes to the same distribution, an agent with "true" logical reasoning abilities can reason with any given information to solve new, unfamiliar problems, regardless of any prior knowledge or different knowledge representations. In other words, it can also generalize across facts, rules and representations (Sec 1 Line 13): 

1. Facts: We test the same rules as in the training but with different facts (Sec 3.2.1 Line 4), as shown in the testing settings in Table 4.
2. Representations: Logical rule reasoning should be independent of different semantic content, including the rep_xx_with_xx examples in Table 4.
3. Rules: We test different first-order rules, such as training on $r1(A,B) \land r2(B) \rightarrow r3(B,A)$ and testing on $r1(A,B) \land r2(B,C) \rightarrow r3(C,B)$, as demonstrated in Fig 3. Different hop rules are also considered as different rules. Relevant settings include pf-sym-depth1, pf-sym-depth2, and Rudas in Table 5.

Our comprehensive experimental results show that LLMs fine-tuned on symbolic logic datasets can achieve generalization in (1) and (2) to some extent with the help of template matching. However, they struggle to generalize to novel rules, emphasizing that fine-tuning LLMs does not achieve generalizable logical reasoning.

Reviewer comment (Evaluation):
All tables and plots: missing error bars.

Author response:
Re W3 (1) and (2): We have included error bars and refined the row names in Tables 4, 5, and 6 to be more descriptive and concise in the revised version of our paper, with the error bars highlighted in red. Thank you for your suggestions.

Re W3 (3): Table 6 represents the performance of RuDaS and FOLIO of the non-fine-tuned LLaMA2-13B-chat model. The results reveal that, although FOLIO's performance surpasses random, it remains inferior to the non-fine-tuned LLaMA2-13B-chat. This further emphasizes our finding that fine-tuned LLMs struggle to generalize to novel rules.

Reviewer comment (Presentation):
Tables 4, 5, 6 can have more informative row names. The current row names are hard to parse.

Author response:
Re W3 (1) and (2): We have included error bars and refined the row names in Tables 4, 5, and 6 to be more descriptive and concise in the revised version of our paper, with the error bars highlighted in red. Thank you for your suggestions.

Re W3 (3): Table 6 represents the performance of RuDaS and FOLIO of the non-fine-tuned LLaMA2-13B-chat model. The results reveal that, although FOLIO's performance surpasses random, it remains inferior to the non-fine-tuned LLaMA2-13B-chat. This further emphasizes our finding that fine-tuned LLMs struggle to generalize to novel rules.

Reviewer comment (Theory):
LLMs can perform simple arithmetics and it couldn't have seen all possible additions / multiplications etc. during training. Doesn't this show it have some ability to learn rules beyond sematics?

Author response:
The review's observation about LLMs performing simple arithmetic tasks raises an intriguing point about their ability to learn rules beyond semantics. However, our research suggests that this ability may not be as robust as it appears. The performance in simple arithmetic tasks could be attributed to data leakage or the application of pre-learned arithmetic solution templates, rather than a genuine understanding or mastery of arithmetic rules.

For instance, when testing GPT-4 with a large number arithmetic problem, such as "128373 + 2879321873=", the model provided an incorrect answer (2879449246 instead of the correct answer 2879450246). This error indicates a limitation in the model's ability to process and apply arithmetic rules, especially in the context of large numbers. In other words, we think if it truly master arithmetics rules (carry or borrow operations), it should process large number. It suggests that while LLMs may display a superficial capability to perform arithmetic operations, they may not truly master the underlying rules.

Therefore, while LLMs exhibit some level of proficiency in arithmetic, this should not be mistaken for a deep or comprehensive understanding of arithmetic rules. Our findings point towards a need for further development in logical reasoning area to enhance the true rule-learning capabilities of LLMs beyond semantic associations.