Paper: Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR-2024)

Reviewer comment (Evaluation):
The results would be more informative if compared directly with human performance. Do any of these benchmarks contain human performance measures (e.g. I believe that there is already human behavioral data for miniSCAN in the original paper)?

Author response:
Thanks for the suggestion! All of our experiments are largely motivated by cognitive scene literature. Therefore, they do have existing human performance as a reference. Note the exact setups, data, and evaluations in these studies might differ from ours. We include all existing human studies results in Appendix C.1 and refer it when discussing main results in Section 3.

Reviewer comment (Evaluation):
The results on noisy rule induction are especially difficult to interpret without a human baseline. The authors cite a paper indicating that humans are somewhat robust to noise when inducing rules, but the amount of noise and the specific tasks will matter a lot.

Author response:
We conducted human study using the same setup and added results in Appendix C.3. We observe that both the LM and humans perform worse on tasks with noisy examples. However, the relative performance drop of the LM is more significant. We clarified this in Section 4.2 footnote 9.

Reviewer comment (Experiments):
Throughout the paper, the authors appeal to intuition concerning putative human performance on the benchmarks they consider (e.g. in considering the potential for human reasoners to show a discrepancy between hypothesis generation and rule application), but intuition is not always a reliable guide regarding human performance. It would be good to qualify these statements a bit more (or, to the extent possible, to include a direct comparison with human performance).

Author response:
Thanks for the feedback! We conducted more human studies to provide a more comprehensive head-to-head comparison between the LM and humans. The additional human experiments include asking crowdworkers to (1) write rules for 50 randomly sampled tasks from List Functions and MiniARC (as a comparison to Table 1), (2) write rules for perturbed tasks with a hint indicating that examples may be incorrect (comparable to the dashed line in Figure 4a), and (3) write rules for perturbed tasks without hints (comparable to the bar Figure 4a). We included all details of these human studies in Appendix C.3. While these studies do not cover all the experiments conducted with LMs due to cost constraints, we hope they provide a better understanding of the behaviors of LMs and humans and motivate future research in this area.

Reviewer comment (Evaluation):
The 'task accuracy' measure seems designed to emphasize the consistency of the symbolic rule application. When looking at raw accuracy, the differences between symbolic and LLM rule application don't look nearly as large. I think this somewhat undermines the claim that their rule application abilities are so much worse than their hypothesis generation abilities.

Author response:
We consider “task accuracy” as an alternative metric as it captures a desirable property of an induction system: it should ideally consistently solve examples within a task. We use this metric to approximate the model’s true understanding of a task because a model is less likely to solve all examples without using the expected computation. Although the differences in raw accuracy are not as large as those in task accuracy, they are still significant for certain tasks (e.g. relative performance drop is around 27% for MiniSCAN and around 42% for MiniARC). In addition, we observe that more advanced prompting techniques, such as self-consistency prompting and zero-shot CoT prompting, also do not bridge the gap (see Appendix B.2 for new experiments). As we argue in Section 4.1, we do not claim that we should expect LMs to behave as effectively as symbolic interpreters, but often the gaps are still large enough to support the argument.

Reviewer comment (Evaluation):
It should be emphasized more that miniARC, unlike the other tasks, is a distinctly visual task, in that it requires understanding visual concepts like 'objectness'. It is somewhat unsurprising that a text-only model would show special difficulty on such a task.

Author response:
Thanks for the suggestions! We added some discussions in Section 3 to better explain the results. Also added explanation in Appendix A dataset details to emphasize that we only use textual representations of the original visual grids.

Reviewer comment (Evaluation):
Do the authors have any explanation for why the hypothesis refinement approach achieves worse performance on miniARC relative to the baselines? Given the visual nature of the task, it's not surprising that it doesn't help much, but I was surprised that it actually seems to impair performance.

Author response:
Many tasks in MiniARC can be solved through pattern matching. In such cases, learning a simple dictionary that maps colors between seen examples and unseen examples may be sufficient (see the bottom example in Table 11 in Appendix B.2). LMs have been shown to be good at pattern matching (Mirchandani et al., 2023). However, generating a rule involves a higher level of abstraction, which could be more challenging for LMs. Importantly, in our setup, the interpreter does not have access to seen examples. Therefore, the rule must contain sufficient and accurate information for the interpreter to achieve strong performance when applying the rule. We explained this in the updated version in Section 3.

Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng. Large Language Models as General Pattern Machines. 2023.

Reviewer comment (Experiments):
The familiarity analysis seems to confound two very different issues: 1) The presence of the same exact problems in the LLM pretraining data (a significant possibility, given the use of pre-existing datasets), and 2) the use of familiar vs. unfamiliar words (e.g. pseudowords). I think it's important to dissociate these concerns. This can be done by creating a new dataset with similar properties, e.g. by replacing the specific pseudowords and colors in miniSCAN (but maintaining the same general pseudoword -> color structure).

Author response:
We share similar concerns. For MiniSCAN, we did not use any existing examples, but instead generated our own examples to mitigate the potential data contamination and uncontrolled lexical exposure issues (see Appendix A for dataset details). The “pseudoword → color” follows the same setup in Lake et al. (2019), but uses different re-generated pseudowords for inputs. The “pseudoword → pseudoword” uses re-rengerated pseudowords for both input and output. We clarified this in the updated version.

For the “pseudoword → color” setup, it is still possible that the output has been seen during pre-training. To eliminate this factor, we experiment a new setup where we replace the color names used in Lake et al. (2019) to other common color names. For instance, the “dax → RED” can become “dax → ORANGE”. We evaluate 100 tasks and show results below. 

|                       Setup | Raw Acc. | Task Acc. |
|-----------------------------|----------|-----------|
| pseudoword → original color |     93.3 |      85.0 |
|      pseudoword → new color |     97.1 |      95.0 |
|     pseudoword → pseudoword |     86.6 |      71.0 |

The “pseudoword → original color” and “pseudoword → new color” setups both achieve higher accuracy compared to the “pseudoword → pseudoword” setup. This is consistent with our claim that LMs’ performance drops when the output representation deviates from their pre-training distribution.

Reviewer comment (Evaluation):
Is the interaction between iid vs. ood and IO vs. hypothesis refinement (figure 2) statistically significant? Also, the text describes the hypothesis refinement results as demonstrating superior robustness to this ood setting, but for miniARC the ood accuracy is actually higher for the IO baseline (even though the difference between iid and ood performance is larger for IO). This should be clarified in the text.

Author response:
We clarified hypothesis refinement results (emphasizing the raw accuracy of IO prompting is still better than hypothesis refinement on MiniARC) in Section 3. 

We added two additional experiments. First, we run paired T-tests on IID vs OOD examples for IO and hypothesis refinement (250 tasks for List Functions and 130 tasks for MiniARC). We show the p-value below. 

|      | List Fns | MiniARC |
|------|----------|---------|
| IO   | 5.9e-19  | 5.4e-4  |
| Rule | 4.2e-7   | 2.5e-2  |

The difference between IID and OOD using IO prompting is much more statistically significant than rule prompting, suggesting that hypothesis refinement is more robust to OOD generalization. 

Additionally, we also run hypothesis refinement (T=3, N=5) for 3 runs and report the mean and standard deviation. We did not run multiple runs for IO prompting, since the results are deterministic when using greedy decoding. The results are shown below. The standard deviation is low across multiple runs. 

List Functions
|  Run | IID Raw Acc. | OOD Raw Acc. | IID Task Acc. | OOD Task Acc. |
|------|--------------|--------------|---------------|---------------|
|    1 |         71.2 |         65.7 |          61.2 |          62.4 |
|    2 |         67.5 |         63.9 |          58.4 |          59.2 |
|    3 |         71.2 |         66.1 |          61.6 |          63.2 |
| Mean |         70.0 |         65.2 |          60.4 |          61.6 |
|  Std |          1.7 |          1.0 |           1.4 |           1.7 |

MiniARC
|  Run | IID Raw Acc. | OOD Raw Acc. | IID Task Acc. | OOD Task Acc. |
|------|--------------|--------------|---------------|---------------|
|    1 |         18.7 |         13.5 |          14.6 |          12.3 |
|    2 |         17.2 |         12.3 |          12.3 |          10.8 |
|    3 |         16.7 |         11.5 |          13.1 |          10.0 |
| Mean |         17.2 |         12.3 |          13.1 |          11.0 |
|  Std |          1.0 |          1.0 |           1.2 |           1.2 |

Reviewer comment (Evaluation):
Were the language model and human hypotheses systematically compared in any way, or only qualitatively inspected? Did the strength of the language model hypotheses correspond to performance on the task? Were there cases where the model performed well on the task despite providing unhuman-like hypotheses?

Author response:
In addition to qualitative evaluation, we ask human annotators to quantitatively rate between LM-induced rules and human-induced rules using two metrics: clarity and supportiveness. See Appendix C for details. For List Functions, where the LM achieves high accuracy, LM-induced rules and human-induced rules are comparably clear, but the former are sometimes less supportive. On MiniARC, where the LM performs poorly, we observe a significant performance gap between LM-induced rules and human-induced rules on both clarity and supportiveness. This seems to suggest that good performance is correlated with similarity with human-rules. We haven't found cases where unhuman-like hypotheses can still achieve very good performance in our experiments.

Reviewer comment (Reproducibility):
What setting was used for the 'top p' parameter in GPT-4?

Author response:
We use default `top_p = 1`.

Reviewer comment (Presentation):
It would be helpful to either include the variable names in figure 1, or to have a separate figure illustrating the overall flow of the model with the corresponding variable names.

Author response:
Thanks for your suggestion! Due to the space limit, we add a sentence in the caption of Figure 1 to clarify the two main hyperparameters (T, N) used in iterative refinement.

Reviewer comment (Novelty):
It would be good to say a bit more about how this work differs from Wang et al (2023). This is concurrent work, so there is no concern about novelty, but it would still be useful to have more discussion of the relationship.

Author response:
Our paper shares a similar spirit with Wang et al. (2023) in exploring the inductive reasoning capabilities of LMs. However, we focus on understanding both the potentials and limitations of LMs. For this reason, our work differs from Wang et al. (2023) from several perspectives. 

- *Methodology-wise*, we explore various domains and different types of interpreters to validate the effectiveness of iterative hypothesis refinement. We minimized task-specific heuristics (e.g. we did not explicitly explain useful visual concepts like “objectness” in MiniARC) and did not exhaustively tune hyper-parameters. We expect better performance by exploring alternative prompt templates and tuning hyper-parameters, as suggested by Wang et al. (2023).
- *Analysis-wise*, we offer more exploration of the limitations of LMs and identify several counterintuitive behaviors of LMs in Section 4. We also conduct human studies (Appendix C) to contrast the inductive reasoning capabilities of LMs with those of humans.

We discuss the relationship between our work and Wang et al. (2023) at the end of Section 1 and in the related work section. We also point out in Section 5 that some techniques used in Wang et al. (2023) could potentially further improve performance.

Reviewer comment (Writing):
I found the description of the approach somewhat confusing. Based on the abstract and intro, I was expecting that the hypotheses would be articulated in natural language, and this would somehow be translated into code which is then symbolically executed. It is explained later on (in section 2.2) that the LM also carries out this translation step for list functions and miniARC, but it would be good to provide some hint that this is the case earlier in the section describing the approach. My understanding is that for the other tasks, the LM is prompted so as to ensure hypotheses in a particular format, which can then be automatically parsed, is that correct? It would be helpful to clarify this (in section 2 it says that the hypotheses are 'constrained', but it's not immediately clear what that means).

Author response:
Yes, for ACRE and MiniSCAN, the LM is prompted to generate hypotheses in a particular format. We clarified this in the introduction to emphasize that the hypothesis could be either free-form (in which case we use an LM to translate the hypothesis to a format interpretable by the symbolic interpreter) or constrained (in which case we can directly pass it to the interpreter). 

Additionally, for List Functions and MiniARC, we also evaluate directly generating Python hypotheses (see Appendix B.2. Representation of Hypothesis). As shown in Table 9, using programming language hypotheses achieve comparable performance to natural language hypothese, suggesting that this representation can be a promising alternative for these tasks.

Reviewer comment (Writing):
I found this sentence, 'However, they also behave as puzzling inductive reasoners, showing notable performance gaps in rule induction (i.e., identifying plausible rules)…' to be confusing, because it seems to say that LLMs are bad at proposing rules, even though it was just stated that they are good at this. This also seems misaligned with the results. It seems like this sentence should instead emphasize the rule application specifically (though, as mentioned above, it's not clear how significant this discrepancy really is).

Author response:
We intend to highlight that there are significant performance gaps between rule induction (good) and rule application (poor). In other words, LMs are much more effective at generating meaningful hypotheses than at applying the rules they propose. We hope this clarifies the confusion.

Reviewer comment (Novelty):
The authors might consider citing this related work on analogical reasoning (a special case of inductive reasoning) in LLMs: Webb, T., Holyoak, K. J., & Lu, H. (2023). Emergent analogical reasoning in large language models. Nature Human Behaviour.

Author response:
Thank you so much for your super useful feedback! We will emphasize your point in the revision and update the abstract based on your suggestion.

Reviewer comment (Evaluation):
Similarly, the complexity of the human induced and LLM induced rules might be interesting to analyze.

Author response:
Since the human-induced and LM-induced rules are often in free-form natural language, it is even more challenging to measure their complexity. Measures such as minimum description length and the number of unique n-grams could be potential solutions. However, it is unclear whether these measures correlate with the complexity of rules in our setup. We agree that this is a very interesting direction, but it is beyond the scope of this paper. Therefore, we leave this as a direction for future work.

Reviewer comment (Experiments):
How were the number of examples seen by the model chosen across domains? What is the minimum number of examples needed to learn a rule?

Author response:
We leverage the existing datasets for our experiments, therefore the number of seen examples for each dataset is given, except for List Functions where we only use 8-shot examples. Our preliminary experiments on List Functions suggest that increasing the number of seen examples does not improve performance. Since our tasks are relatively self-defined, a small number of seen examples is generally sufficient. Finding the minimum number of examples necessary for induction is definitely an interesting and ongoing research direction! However, as we noted in footnote 11, this is out of the scope of this paper.

Reviewer comment (Reproducibility):
An open source model would make the evaluations more comprehensive.

Author response:
We included LLaMA2-70B results in Appendix B.1.

Reviewer comment (Experiments):
A separate evaluation for LLMs as symbolic interpreters of rules would help tease apart the rule-proposing / application components more. More on complexity: LMs might be bad appliers of complex rules.

Author response:
We include this analysis in Section 4.1. In Figure 3, we compare the accuracy when applying the rules using symbolic interpreters or the LM itself as the interpreter. We observe a consistent performance drop when using the LM interpreter instead of the symbolic interpreter. This suggests that while LMs are effective at proposing rules, they may not be able to apply their own proposed rules.

Reviewer comment (Experiments):
Is there a change in the types of rules induced if the prompt is changed to encourage communication (since this was what humans seemed to do)? Change prompt to emphasize communication?

Author response:
First of all, to investigate whether other prompts would lead to different behaviors, we evaluate an alternative hypothesis generation prompt that includes task-specific heuristics (see Appendix B.2, Task-specific Heuristics). Specifically, we use MiniARC and add a detailed task description in the prompt (Table 10). This does help LMs to generate more human-readable rules (Table 11) for a fraction of examples, but these rules are still generally distinguishable from those induced by humans. Additionally, we did not observe performance improvement by imposing these constraints. While it is possible that other prompts could encourage LMs to communicate more, this would require additional prompt engineering and is beyond the scope of this paper. We mention this as a potential direction for future work.

Secondly, for a fair comparison, we also did not instruct annotators to use any communication strategies nor did we provide any constraints or hints on the types of rules (see annotation interface in Figure 8). Humans appear to naturally use pragmatic communication strategies. Our point is that, without any specific instruction, LMs demonstrate inductive reasoning behaviors that are different from those of humans. Emphasizing communication could certainly be useful from a practical standpoint, but it is not our main focus in understanding the limitations of LMs in inductive reasoning.

Reviewer comment (Experiments):
For the example perturbation experiment in section 4.2, there are no studies on how well humans can actually perform on perturbed tasks. It is hard to judge how big the performance drop of LMs is compared with humans.

Author response:
Thanks for the feedback! We conducted human study using the same setup and added results in Appendix C.3. We observe that both the LM and humans perform worse on tasks with noisy examples. However, the relative performance drop of the LM is more significant. We have clarified this in Section 4.2 footnote 9.

Reviewer comment (Experiments):
The generalizability of the findings in 4.3 is doubtful because only one type of prompt is used to generate rules from LM. According to the appendix, example rules on List Fn and MiniARC are generated from LM with a prompt that contains no format instruction. It is unknown whether LMs can generate human-like inductions with more guidance or provided with human-induced rules as few shot examples.

Author response:
We did not introduce task-specific format instructions for List Functions and MiniARC, as they cover a wide range of concepts, making it challenging to determine the most optimal constraints. To investigate whether other prompts would lead to different behaviors, we evaluate an alternative hypothesis generation prompt that includes task-specific heuristics (see Appendix B.2, Task-specific Heuristics). Specifically, we use MiniARC and add a detailed task description in the prompt (Table 10). This does help LMs to generate more human-readable rules (Table 11) for a fraction of examples, but these rules are still generally distinguishable from those induced by humans. Additionally, we did not observe performance improvement by imposing these constraints. It is certainly possible that other guidance or few-shot examples could encourage LMs to generate human-like rules. However, the former requires more prompt engineering, and the latter requires additional human annotations, which are beyond the scope of this paper. We mention these alternatives as potential directions for future work.

Reviewer comment (Reproducibility):
What are the prompts used to generate the Python program from rules for List Functions and MiniARC?

Author response:
We use the following prompt template:

“You are an expert Python programmer. Write a Python function `fn` for the following rule. {Example description}

Rule: {Rule}”

The {Example description} describes the input and output types, e.g. a list of integers (List Functions) or a 2D grid of integers (for MiniARC). We show the hypothesis translation prompt in Table 14 in Appendix D.

Reviewer comment (Evaluation):
In Table 2, on MiniScan, T=3, N=1 yields higher raw accuracy and task accuracy than T=3, N=5, which means that fewer hypotheses considered lead to worse performance. Is there an explanation for this?

Author response:
One possible explanation for these results could be the differences in sampling temperature. For N=1, we use greedy decoding (temperature = 0), and for N=5, the temperature is set to 0.7. An increased temperature might lead to more inaccurate hypotheses, similar to what is observed with SC prompting compared to IO prompting.

More importantly, the MiniSCAN dataset is designed to evaluate compositional generalization: certain unseen examples require the novel composition of known concepts. The distribution shift between seen examples and unseen examples means that high accuracy on seen examples does not necessarily translate to high accuracy on unseen examples. For instance, the rule “dax fep → RED RED RED” is less preferable, even if it perfectly explains the given example. Since iterative refinement only uses accuracy over seen examples as the scoring function, it might overfit to seen examples and select hypotheses that are less generalizable. We clarified this in Section 3.

Reviewer comment (Experiments):
In Section 2, the authors claim to evaluate models on “OOD” examples by generating longer or larger examples than those in the original datasets. This is a bit confusing. Are authors actually fixing the seen examples and only changing the unseen examples for testing?

Author response:
Yes. We clarified this in the updated version.

Reviewer comment (Reproducibility):
When asking LMs to write Python programs given the hypothesis, are the seen examples also provided in the prompt?

Author response:
No, we do not provide examples. See Table 14 in Appendix D for hypothesis translation prompt.