Paper: Large Language Models can Learn Rules (ICLR-2024)

Reviewer comment (Reproducibility):
Many details are missing, e.g., how does the induction stage work? How to use ground truth answers to verify the induced rules, does it require human annotators? What is the confidence of rules, how are they evaluated, and does LLM output a confidence score associated with the rules?

Author response:
A1: The induction stage uses an LLM and CoT prompt to generate intermediate steps containing rules and the final answer, which are verified based on the ground truth answer. To compare the generated output and the ground truth answer, we take the last sentence of the generation and check if the ground truth answer is a full word in the output, following the open-source implementation of CoT[1]. As pointed out in Section 3.1, the confidence of a rule is defined by the probability that it leads to the correct answer on the training set. For example, if a rule occurs 5 times across the entire training set, and results in the correct answer for 3 times, its confidence is 0.6. The LLM doesn’t need to output any confidence score for the rules. All the confidence scores are computed by the verification results on the final answer.

Reviewer comment (Theory):
This work is basically a technical report, some claims lack supportive facts or resources. For example, the authors say "hallucination in LLMs resembles hypothesis generation in scientific discovery", which is incorrect. In scientific discovery, hypotheses are generated by logical induction and abduction ([reference](https://plato.stanford.edu/entries/scientific-discovery/)), while the logic behind hallucination remains unknown.

Author response:
A2: We agree that scientific discovery may be generated by logical induction and/or abduction in many cases, and this is likely to be different from the mechanism of hallucination. Here we don’t want to make any claim, but try to draw parallels between the hypotheses and hallucination – both may contain useful information but must be verified on a set of samples. This serves as a motivation for our method. If this doesn’t address your concerns, we are happy to further discuss this and modify the introduction accordingly.

Reviewer comment (Experiments):
Rules in the form of natural language weaken the generalisation ability and usually cause ambiguity and may confuse people. This is exactly the reason why Gottfried Leibniz calls for mathematical logic. However, most of the time, humans, like LLMs, are using natural language for reasoning, so I don't think the method proposed by this paper is bad. It would be interesting to make a more comprehensive test for the learned rules using formal methods, for example, ask LLMs to abstract those numerical rules in Appendix C into higher-order forms, such as Peano axioms and see if LLMs can make use of those more advanced rules.

Author response:
A3: That’s a good question. We tried to prompt an LLM to automatically discover some more abstract rules or principles, but found the model can only do so when we provide the term “Peano axioms” in the instruction, which is too handcrafted for the arithmetic dataset and doesn’t generalize to new tasks. Instead, we use a more generic instruction to ask the LLM to write some knowledge describing the rules. Due to the large variance in the generated description, we sample 5 descriptions for each base system, and report the mean and variance of the results. We found that the description generated by the LLM from the learned rule library can improve the reasoning performance compared to standard CoT, yet it is worse than HtT since the description often loses some detailed information of the rule library. One advantage of the generated description is that it compresses the rule library into some compact descriptions, which reduces the length of the prompt. We will include these results in the appendix of the paper.

| **Arithmetic** | **2 digits** | **3 digits** | **4 digits** | **2 digits** | **3 digits** | **4 digits** | **2 digits** | **3 digits** | **4 digits** | **Avg** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 5-shot CoT | 83.3 | 71.1 | 61.5 | 52.8 | 47.4 | 46.2 | 75.0 | 36.8 | 42.3 | 57.4 |
| 5-shot CoT+HtT | 100 | 94.7 | 84.6 | 88.9 | 71.1 | 46.2 | 86.1 | 68.4 | 65.4 | **78.4 (+21.0)** |
| 5-shot CoT+higher-order forms | 81.6±6.2 | 67.7±8.6 | 60.6±10.2 | 79.5±8.1 | 61.5±11.4 | 62.8±6.7 | 81.6±4.4 | 65.4±8.1 | 55.0±6.9 | 68.4±2.4 (+11.0) |

Reviewer comment (Novelty):
The examples as shown in appendix are not very impressive, at least from a skim through, it seems the rules already exist explicitly in the prompt text. At this point how is this different from just knowledge retrieval as done in [1]. In fact the tasks presented in [1] seem much more nuanced than the one presented here.

Author response:
**W2: The examples as shown in appendix are not very impressive, at least from a skim through, it seems the rules already exist explicitly in the prompt text. At this point how is this different from just knowledge retrieval as done in [1]. In fact the tasks presented in [1] seems much more nuanced than the one presented here.**

A2: We guess you mistook the rules in HtT prompts as human-written or code-generated, while they are actually learned by LLMs in the induction stage. We’ve clarified that rules in HtT prompts (Prompt 3, 5, 8 & 10) are just for demonstrating the format of the prompts in Appendix B. All the rule libraries used by HtT (Prompt 13-16) are learned in the induction stage. The only human-written rules are those in the few-shot exemplars, which are less than 10% of the size of the learned rule libraries for every dataset we used. Note the same set of human-written rules are used in the few-shot CoT prompt. So it is fair to use them to elicit and learn rules in HtT.

IRCoT[2] augments CoT with an external corpus and a pretrained retriever, which is totally different from the setting of HtT. HtT learns a rule library from a training set and a standard CoT prompt, without any external corpus. Once the rule library is learned, the deduction stage of HtT is similar to the setting of IRCoT. We agree that IRCoT may be a better solution for our deduction stage, but we implement the rule library and retrieval steps as a prompt to make it a fair comparison with CoT.

[1] Karthik Valmeekam, et al. On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark). arXiv 2023.

[2] Harsh Trivedi, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv 2022.

Reviewer comment (Experiments):
The gains without XML tagging (an existing method in the prompting technical know-how) are marginal. Furthermore, (it seems to me) that the authors have not tested CoT and LtM with XML tagging, making it unclear how much of their gains are from tagging, and how much is from the extracted knowledge.

Author response:
**W3 & Q1: The gains without XML tagging (an existing method in the prompting technical know-how) are marginal. Furthermore, the authors have not tested CoT and LtM with XML tagging, making it unclear how much of their gains are from tagging, and how much is from the extracted knowledge.**

A3: We agree that XML tagging brings a large gain in the performance, as shown by the ablation studies in Table 4. We are aware that XML tagging is used to separate different sections of the input, as documented by OpenAI and Anthropic. Here we extend this technique to guide the retrieval of knowledge in LLM generation. The XML tagging provides a prompting solution to improve the retrieval ability of LLMs without finetuning the model on retrieval tasks. As far as we know, there is no existing paper using XML tagging in such a way. With that being said, if you know any paper regarding this trick for retrieval, we are happy to cite and credit it.

This is a good suggestion on ablation studies. Here we apply the same XML tagging trick to CoT and LtM. Note that there isn’t any rule library in CoT and LtM, so the tags only help elicit rules from the LLM, rather than retrieve rules from the library. We observe that XML tagging doesn’t contribute significantly to the performance of our baseline methods except for CoT on Arithmetic. For both datasets and both prompting methods, HtT consistently outperforms baseline methods with XML tagging by a large margin, which suggests that our performance gain comes from the synergy of the learned rule library and XML tagging, rather than XML tagging alone. We will add these results in the revision.

| **Prompt** | **Arithmetic** | **CLUTRR** |
| --- | --- | --- |
| 5-shot CoT | 57.4 | 50.6 |
| 5-shot CoT (w/ XML tags) | 64.0 (+6.6) | 47.7 (-2.9) |
| 5-shot CoT + HtT (w/ XML tags) | **78.4 (+21.0)** | **64.3 (+13.7)** |

| **Prompt** | **Arithmetic** | **CLUTRR** |
| --- | --- | --- |
| 5-shot LtM | 51.0 | 46.8 |
| 5-shot LtM (w/ XML tags) | 52.6 (+1.6) | 36.4 (-10.4) |
| 5-shot LtM + HtT (w/ XML tags) | **62.0 (+11.0)** | **57.9 (+11.1)** |

Reviewer comment (Experiments):
The examples that you show in appendix seem very simple rules, are you able to extract complex rules? — beyond "A is B".

Author response:
**W1 & Q2: The setting is too simple, the learnt rules are just true/false propositions of the form "A is B".**

A1: The goal of using a simple setting is to reduce inference cost and allow manual analysis of error cases (e.g. Figure 3 in the paper). However, we should emphasize that HtT is not limited to learning proposition logic rules. Here we experiment HtT with first-order logic rules (e.g. If Y is X's aunt and Z is Y's father, Z is X's grandfather) on CLUTRR. It is observed that HtT with first-order logic rules perform even better than the propositional logic rules. We will update these results in the revision.

We note that HtT is not limited to certain forms of rules. However, besides rule learning, another problem that is important but is beyond the scope of this paper is the planning problem. In general, reasoning with first-order logic rules requires unification (i.e. to match rules and facts at each step), where for CLUTRR dataset the reasoning is a path-based special case. Empirically, we find it is hard to design a CoT prompt that teaches LLMs to perform unification for general case, as it requires planning abilities that LLMs typically don’t possess[1]. As a result, we can’t learn useful rules based on the CoT prompt when the task requires unification. A future work is to empower the rule learning with planning. If we have a deductive reasoning prompt that can perform unification, we believe HtT with that base prompt should be able to learn rules in general first-order logic settings.

| **CLUTRR** | **2 hops** | **3 hops** | **4 hops** | **5 hops** | **6 hops** | **7 hops** | **8 hops** | **9 hops** | **10 hops** | **Avg** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0-shot CoT | 50.0 | 22.2 | 22.6 | 32.0 | 37.5 | 38.1 | 33.3 | 46.2 | 16.0 | 33.1 |
| 5-shot CoT (propositional) | 50.0 | 55.6 | 71.0 | 80.0 | 50.0 | 52.4 | 30.0 | 46.2 | 20.0 | 50.6 |
| 5-shot CoT+HtT (propositional) | 100.0 | 61.1 | 74.2 | 84.0 | 75.0 | 38.1 | 56.7 | 53.8 | 36.0 | 64.3 (+13.7) |
| 5-shot CoT (first order) | 50.0 | 71.4 | 90.9 | 90.6 | 76.5 | 88.6 | 75.9 | 66.7 | 66.7 | 75.2 |
| 5-shot CoT+HtT (first order) | 100.0 | 71.4 | 97.0 | 96.9 | 82.4 | 85.7 | 93.1 | 88.9 | 83.3 | **88.7 (+13.5)** |

Reviewer comment (Experiments):
Although the method presented in the article exhibits a certain degree of innovation, its articulation fails to meet the standards of ICLR, leaving many details unaddressed within the paper. This omission results in confusion among readers trying to grasp the intricacies of the proposed approach. For instance, in the "Induction from Deduction" section, it is not specified how the rules are extracted — is it through regular expressions? It is also unclear how the occurrence k and accuracy p are calculated based on the paper. These concepts are borrowed from the field of rule learning, yet the author does not elucidate how they are applied in the domain of natural language. This transition from rule learning to natural language processing necessitates a clear explanation, as the methodologies and challenges inherent to these domains can be vastly different. In the appendix, considering Prompt 2, it is evident that a substantial number of ground rules are already present within the prompt. This raises a question: If we rely solely on the rules from the prompt, what level of performance can be achieved?

Author response:
A1: As discussed in Appendix A, we extract the rules with regular expressions. Here are the regular expressions we used:
Numerical reasoning: `[A-Z0-9]+(?: \+ [A-Z0-9]+)+ = [A-Z0-9]+\.`, which accepts a summation equation of arbitrary numbers or characters.
Relational reasoning: `([a-zA-Z\-]+)'s ([a-zA-Z\-]+) is ([a-zA-Z\-]+)\.`, which accepts a sentence describing the combination of two arbitrary words. We note that HtT is not sensitive to the choice of regular expressions, as long as the regular expressions can extract rules from the human-written few-shot exemplars with 100% precision and recall. If the regular expression recalls wrong rules from the LLM, they can be easily filtered by our minimal occurrence criterion.

The occurrence of a rule is just the number of its occurrences in the induction stage. The accuracy of a rule is the probability of reaching the correct answer when the rule occurs. Taking the rules learned in base-9 as an example, a correct rule “4 + 3 = 7” occurs 20 times and leads to correct answers for 9 times in the induction stage, resulting in an accuracy of 0.45. The reason why correct rules don’t have 100% accuracy is that they are sometimes combined with other wrong rules in some examples. A wrong rule “7 + 5 = 12” occurs 42 times and never leads to correct answers in the induction stage, resulting in an accuracy of 0. Therefore, a minimal accuracy p of 0.3 is enough to distinguish these rules. The minimal occurrence k is used to filter less reusable rules such as “5 + 1 + 2 = 8”, which is correct but only occurs when the previous step generates a carry of 2 by mistake. The choices of $k$ and $p$ for different datasets are provided in Appendix A.

Reviewer comment (Presentation):
The citation is inconsistency. For instance, the first referenced NeurIPS paper does not include page numbers, while the second one does. The third citation is missing its source of publication, and the fourth one includes the conference name’s abbreviation, unlike the others. Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems, 2020. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. 2023. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418–5426, 2020.

Author response:
A2: Prompt 2 is the CoT baseline, of which the results have already been provided in Table 1. The same prompt is also used in the induction stage of HtT to learn rules. We copy the results of GPT-4 here for visibility.

| **Arithmetic** | **2 digits** | **3 digits** | **4 digits** | **2 digits** | **3 digits** | **4 digits** | **2 digits** | **3 digits** | **4 digits** | **Avg** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0-shot CoT | 72.2 | 26.3 | 7.7 | 22.2 | 10.5 | 3.8 | 30.6 | 34.2 | 23.1 | 25.6 |
| 5-shot CoT | 83.3 | 71.1 | 61.5 | 52.8 | 47.4 | 46.2 | 75.0 | 36.8 | 42.3 | 57.4 |
| 5-shot CoT+HtT | 100 | 94.7 | 84.6 | 88.9 | 71.1 | 46.2 | 86.1 | 68.4 | 65.4 | **78.4 (+21.0)** |

We note that Prompt 2 provides only a very small fraction of the ground truth rules (10 rules compared to 512/242/162 ground truth rules in base-16/11/9 respectively). It is possible that these rules somehow help the LLM to recall rules, which explains why 5-shot CoT is better than 0-shot CoT. However, as shown in Table 1 & 2, with the learned rule library, HtT significantly improves over CoT.

Reviewer comment (Writing):
The paper falls short of providing a comprehensive description regarding the spectrum of problems that the introduced approach is adept at solving. This leaves a ambiguity as to whether the proposed method can effectively tackle all varieties of hallucination issues, a matter that necessitates further elucidation for a complete understanding of the method’s capabilities and limitations.

Author response:
A3: Thanks for pointing that out. We will update it in the revision.

Reviewer comment (Theory):
Furthermore, is the method capable of learning complex rules, such as first-order logic rules? If unification is required during reasoning, can LLMs still utilize these rules for inference?

Author response:
A4: We plan to add a paragraph discussing the scope of HtT in the camera ready version. Based on what we observed for HtT in experiments, it has the following constraints.

1) The task is approachable via CoT prompting, where each intermediate step explicitly uses a rule.
2) While the rules can be in natural languages, they should have a consistent template, otherwise it is hard to count the occurrences of rules of the same semantics. 
3) The same set of rules are applicable for every sample in the whole dataset, otherwise we can’t reuse rules learned from the training samples on the test samples. 
4) CoT has a reasonable performance on the task, otherwise we often get errors in the induction stage and can’t learn the rules. 
5) The size of the rule library is moderately small, otherwise we can’t fit it into the context length of the LLM in the induction stage.

We believe only constraint 3 is a hard limitation on the scope of problems solvable via HtT compared to CoT. Constraint 2 may be improved by using external models for matching rules of the same semantics. Constraint 4 is a general drawback on CoT. For example, CoT fails on tasks requiring unification of first-order logic rules, and we can’t apply HtT in this case either. Constraint 5 may be solved by offloading the learned rule library to external databases using RAG or equivalent methods.