Paper: Hypothesis Search: Inductive Reasoning with Language Models (ICLR-2024)

Reviewer comment (Evaluation):
It would be nice if there a statistical analysis on the failure case analysis.

Author response:
Thanks for the excellent suggestion! For the human-selected evaluation, we’ve now added a simple failure case analysis - namely, we were curious about the following: for what subset of the questions did the model fail because it failed to generate a correct hypothesis, and for what subset did it fail because it failed to implement a correct hypothesis? We looked at the 100 sampled tasks and the human-selected correct generated hypotheses (though, we reiterate here that “correct” is both subjective and somewhat arbitrary without experiments in line with those conducted in LARC). For 49% of them, at least one correct hypothesis was identified as correct. Of those selected, 33 (67%) led to correct programs. However, when given all correct language (i.e., the Human-Written Hypothesis evaluation), the model successfully implements 45% correctly. Of these, only one task is solved by the human-selected pipeline that isn’t solved by the human-written pipeline. This may suggest that there is a relationship between the tasks that the model can propose a hypothesis for and the tasks for which it can implement that hypothesis.

Reviewer comment (Novelty):
The overall prompting framework in this paper is very similar to self-debug[1], except that self-debug focuses on iterative refinement, while this paper emphasizes hypothesis search. If this is the point, the authors should provide a deeper analysis of the generated hypotheses. Algorithm 1 has a similar high-level idea of Figure 3 from the self-debug paper. So this paper is more like revisiting self-debug from a different perspective, which limits its novelty and contribution. This paper also misses an important citation[2].

Author response:
> The overall prompting framework in this paper is very similar to self-debug[1] , except that self-debug focuses on iterative refinement, while this paper emphasizes hypothesis search.

First, we apologize for the misunderstanding. Self-debug does not explore inductive reasoning (that is, taking a collection of examples and identifying their underlying rule). Not only does it not “emphasize hypothesis search” - indeed, hypothesis search is not part of their method, and is our primary contribution.

In particular, we are (to our knowledge) the first to demonstrate that language models can effectively generate a collection of hypotheses to solve an inductive reasoning problem, and then select the right hypothesis by evaluating its ability to explain the inductive reasoning problem’s examples. We have added a discussion to the Introduction section to help clarify this.

While our paper does use the language model to revise its solutions in code, we also strongly agree that this idea is not one of our contributions. We have updated the paper to more clearly emphasize that our primary contribution is programmatic inductive reasoning pipeline via hypothesis generation, and have added additional citations to prior work on automated code repair [1,2,3,4] to help contextualize this work.

[1] "A Bidirectional LSTM Language Model for Code Evaluation and Repair" Rahman et al 2021  
[2] "Neural Program Repair by Jointly Learning to Localize and Repair" Vasic et al 2019  
[3] "Deep Learning for Code Repair" Pang 2018  
[4] "Automated program repair through the evolution of assembly code" Schulte 2010

Reviewer comment (Experiments):
Experiments results are not sufficient to justify the significance of the method. Of the 3 datasets used in the paper, the proposed method only works on ARC and 1D-ARC, which are very similar. Besides, it only uses 40 samples for inference and the variance of the performance is not reported. It is likely the observation in this paper may be overestimated due to variance in performance and model selection.

Author response:
> Besides, it only uses 40 samples for inference and the variance of the performance is not reported. 

Thanks for this great point. Originally, scaling this up was prohibitively expensive. However, we have now been able to access additional resources, allowing us to extend these results to 100 tasks for ARC, 1D-ARC and the new List Functions datasets. All the conclusions remain consistent with the previous results, which demonstrate the effectiveness of our proposed pipeline.

Reviewer comment (Novelty):
The contribution of this paper is not very clear. From the intro, it looks like the authors try to solve the inductive reasoning problem. From the experiments, there is no comparison with non-LLM baselines, and it looks more like an ablation study of using natural language hypotheses in program synthesis.

Author response:
> From the intro, it looks like the authors try to solve the inductive reasoning problem. From the experiments, there is no comparison with non -LLM baselines, and it looks more like an ablation study of using natural language hypotheses in program synthesis.

Thank you for highlighting this aspect of our work. First we want to note that on SyGuS, we have reported the accuracy of a non-LLM baseline CrossBeam [5], which is outperformed by our methods. In our paper, we specifically concentrate on exploring inductive reasoning through the lens of language models, as indicated in our title. This focus is intentionally chosen to better understand the capabilities of LLMs in this context. Our main focus was on understanding the impact of generating various kinds of intermediate hypotheses when solving inductive reasoning tasks with language models, a type of reasoning that they have been known to struggle with. For some context, we’ve also now highlighted the special-purpose DSL-based state-of-the-art on ARC (according to [6]) in our text, which performs marginally worse than our results with human-written hypotheses, solving 43 of our 100 sampled tasks.

Reviewer comment (Theory):
Is there any deeper connection between Hypothesis Search and the Bayesian learner mentioned in the introduction?

Author response:
> Is there any deeper connection between Hypothesis Search and the Bayesian learner mentioned in the introduction? 

Under the assumption of a low false-positive rate, which we empirically observed, our method is approximately the same as importance sampling in a hierarchical Bayesian model, where the importance distribution is that of the language model conditioned on the examples. This could potentially guide future work toward more efficient algorithms -- we’ve added a note about this to future work.

Reviewer comment (Experiments):
Sec. 3.2.3. How about the ability of GPT3.5 in generating hypotheses? Why is there no table for this section?

Author response:
> How about the ability of GPT3.5 in generating hypotheses? 

For ARC we noted that we observed GPT-3.5 to mostly generate meaningless hypotheses. However, motivated by this question, we ran additional experiments to understand the performance of GPT3.5-generated hypotheses on 1D-ARC, SyGuS and List Functions when using GPT3.5-generated hypotheses. Notably, on all of the datasets we evaluated besides ARC, natural language hypothesis search improved performance over both a direct prompting baseline, as well as a program-only hypothesis search. Surprisingly, in the case of 1D-ARC, the program-only GPT-3.5 results were actually slightly worse than its direct prompting results, while the full hypothesis search pipeline was better than both. We have added the following table to Appendix B.

|                    | 1D-ARC | SyGuS | List Func |
|--------------------|--------|-------|-----------|
| Direct             | 25.9   | N/A   | 16        |
| Program Only (Ours)| 23.1   | 80.9  | 46        |
| Full (Ours)        | 26.9   | 86.5  | 57        |

Reviewer comment (Presentation):
Sec. 3.4. Why is there no table for this section? Also the last sentence is an overclaim. It’s the improvement of GPT-4 over CrossBeam, not the proposed prompting technique.

Author response:
>Sec. 3.4. Why is there no table for this section? 

We have updated the paper to summarize the results in a table.

> Also the last sentence is an overclaim. It’s the improvement of GPT-4 over CrossBeam, not the proposed prompting technique.

We agree that the results are not directly comparable. We have revised the text accordingly. We provide the number for CrossBeam as for a reference of the state-of-the-art DSL-based method.

Reviewer comment (Novelty):
The technical novelty is limited and the main challenge of generating high-quality programs is largely unsolved.

Author response:
> The main challenge of generating high-quality programs is largely unsolved.

We agree that there is still substantial work to be done before these difficult inductive learning tasks can be considered complete. On the other hand, our methods achieve large improvement over four datasets, which we believe is substantial. Our main contribution is showing how to approach inductive learning using program generation as a component, instead of tacking the program generation problem. Future improvements on generating high-quality programs can be expected to benefit our approach to induction reasoning tasks.

>The technical novelty is limited

Novelty is, of course, a subjective quality. Some technical novelty derives from the extensive mathematical reasoning required. Other novelty derives from simple ideas which haven’t been previously noticed and yet have powerful implications. We believe our paper is an example of the latter. The ability to do inductive learning from a few examples has been a cornerstone ability of LLMs in recent years, and their failure in more complex inductive learning tasks (e.g. ARC) has been taken by a substantial subset of computer scientists as a condemnation of the enterprise. Thus finding that a structured approach to induction yields very large improvements, and that this depends on the right staging between natural and formal languages, is an important finding. To those who expected LLMs to make no progress on hard problems of induction it is a surprising, we’d say novel, one.

Reviewer comment (Evaluation):
The effectiveness of the proposed method of using hypotheses to guide program generation has unclear applicability. (1) GPT-3.5 fails to generate meaningful hypotheses. (2) GPT-4 hypotheses are not helpful on SyGuS where GPT-4 can directly generate good programs. (3) GPT-4 hypotheses are helpful on ARC, but ARC results are still only 37.5 with the hypotheses. Practitioners will have to develop alternative models that can better understand 2D geometry to solve the task and then natural language hypotheses may no longer be helpful as in SyGuS. (4) Model-generated hypotheses hurt the performance of Parsel, a compositional program generation method that can significantly improve the performance when model-generated hypotheses are not used.

Author response:
> The effectiveness of the proposed method of using hypotheses to guide program generation has unclear applicability. (1) GPT-3.5 fails to generate meaningful hypotheses. (2) GPT-4 hypotheses are not helpful on SyGuS where GPT-4 can directly generate good programs. (3) GPT-4 hypotheses are helpful on ARC, but ARC results are still only 37.5 with the hypotheses. Practitioners will have to develop alternative models that can better understand 2D geometry to solve the task and then natural language hypotheses may no longer be helpful as in SyGuS. (4) Model-generated hypotheses hurt the performance of Parsel, a compositional program generation method that can significantly improve the performance when model-generated hypotheses are not used.

Thanks for these questions! We’ll respond to them one-by-one:
1. While GPT-3.5 underperforms GPT-4 (which should not be surprising), given the far lower cost, as well as the recent release of another, better turbo model, this difference should continue to decrease with time. However, motivated by this question, we ran additional experiments to understand the performance of GPT3.5-generated hypotheses on 1D-ARC, SyGuS and List Functions when using GPT3.5-generated hypotheses. Notably, on all of the datasets we evaluated besides ARC, natural language hypothesis search improved performance over both a direct prompting baseline, as well as a program-only hypothesis search. Surprisingly, in the case of 1D-ARC, the program-only GPT-3.5 results were actually slightly worse than its direct prompting results, while the full hypothesis search pipeline was better than both. We have added the following table to Appendix B.
2. Please refer to our response to your first question above.
3. We agree that our method cannot achieve perfect accuracy on ARC, a dataset notably well-known for its difficulty. We believe our proposed pipeline is a first step towards using language models for solving related reasoning tasks, which beats the baseline method by a large margin on ARC. Future work can explore the directions of generating better hypotheses or generating better programs from language hypotheses.
4. We agree Parsel does not perform optimally with our existing pipeline. It is important to note that Parsel was initially developed specifically for solving competitive programming problems from human written descriptions. The mismatch seems to indicate differences in the decomposability of human and model-generated hypotheses, which is an interesting avenue for future work. However, our observations shouldn’t be over-interpreted, as they may be as much about quirks in Parsel and in the current system.

*Table: Additional GPT-3.5 Hypothesis Generation Results*
| Method             | 1D-ARC | SyGuS | List Func |
|--------------------|--------|-------|-----------|
| Direct             | 25.9   | N/A   | 16        |
| Program Only (Ours)| 23.1   | 80.9  | 46        |
| Full (Ours)        | 26.9   | 86.5  | 57        |

Reviewer comment (Writing):
Typo: Sec 3.1 "It contains Although simpler..."

Author response:
> Typo: Sec 3.1 "It contains Although simpler..."
Thank you, this is now fixed!

Reviewer comment (Evaluation):
Sec 3.2.2 says summarized hypotheses can often become vague and ambiguous. Will the hypotheses used to guide program generation be of higher quality if you let the model rank the hypotheses? You could analyze the recall@k, i.e., whether top k hypotheses contain a correct one.

Author response:
> Sec 3.2.2 says summarized hypotheses can often become vague and ambiguous. Will the hypotheses used to guide program generation be of higher quality if you let the model rank the hypotheses? You could analyze the recall@k, i.e., whether top k hypotheses contain a correct one.

We have an experiment very similar to this, currently in Appendix B.2 under the paragraph titled “Using LLMs to Rank Hypotheses.” Unfortunately, we were disappointed in the LLM’s ability to rank hypotheses. We specifically followed the approach from “Coder reviewer reranking for code generation” by [2].

Reviewer comment (Evaluation):
ARC: In Table 2, using human written hypotheses only has 37.5 accuracy. Does that mean LLM fails to write programs based on correct hypotheses? The statement at the end of page 5 that "GPT-4 is pretty good at both generating hypotheses and realizing them as programs" requires some more evidence or explanation.

Author response:
> ARC: In Table 2, using human written hypotheses only has 37.5 accuracy. Does that mean LLM fails to write programs based on correct hypotheses? The statement at the end of page 5 that "GPT-4 is pretty good at both generating hypotheses and realizing them as programs" requires some more evidence or explanation.

First, we agree: we have removed this sentence. For some context, this was intended as a relative statement, given the poor performance of the LLM-based alternatives. We have also added a more explicit we’ve now added a failure case analysis - namely, we were curious about the following: for what subset of the questions did the model fail because it failed to generate a correct hypothesis, and for what subset did it fail because it failed to implement a correct hypothesis? We looked at the 100 sampled tasks and the human-selected correct generated hypotheses (though, we reiterate here that “correct” is both subjective and somewhat arbitrary without experiments in line with those conducted in LARC). For 49% of them, at least one correct hypothesis was identified as correct. Of those selected, 33 (67%) led to correct programs. However, when given all correct language (i.e., the Human-Written Hypothesis evaluation), the model successfully implements 45% correctly. Of these, only one task is solved by the human-selected pipeline that isn’t solved by the human-written pipeline. This may suggest that there is a relationship between the tasks that the model can propose a hypothesis for and the tasks for which it can implement that hypothesis.

Reviewer comment (Evaluation):
For ARC, why do you only consider top-1 accuracy but not top-3 as in the official evaluation? Can you compare your method with state-of-the-art methods on the task?

Author response:
> For ARC, why do you only consider top-1 accuracy but not top-3 as in the official evaluation? Can you compare your method with state-of-the-art methods on the task?

Although top-3 is indeed technically the “official” evaluation metric, top-1 has been the often-unstated convention in LLM-based papers (e.g. [3,4,5]). In general, we would argue that pass@1 makes more sense since we care about evaluating generalization. For some context, we’ve also now highlighted the special-purpose DSL-based state-of-the-art (according to [4]) in our text, which performs marginally worse than our results with human-written hypotheses, solving 43 of our 100 sampled tasks.

Reviewer comment (Experiments):
What are the types of tasks that (1) program generation and (2) hypotheses search can be helpful? Can you summarize the features of such tasks? "Inductive reasoning tasks" is too general and abstract. To begin with, is it true that the method is applicable only to symbolic pattern recognition tasks?

Author response:
> What are the types of tasks that (1) program generation and (2) hypotheses search can be helpful? Can you summarize the features of such tasks? "Inductive reasoning tasks" is too general and abstract. To begin with, is it true that the method is applicable only to symbolic pattern recognition tasks?

In principle, we would argue this framework should be applicable for any inductive reasoning task that can be represented computationally — in practice, there are a few types of inductive reasoning tasks that are not currently supported in our framework. 

For example, many more complex visual inductive reasoning tasks will require extending this implementation, such as using a compositional visual programming language to represent the program (e.g., VISPROG [6]). Moreover, if a task is truly unprogrammable, then we lose a core value of this work, namely the inductive bias and generalization that comes with a programmatic representation. Some challenges may also arise for models with a higher false-positive rate which we’ve now noted in the limitations. 

In terms of tasks that cannot be solved by Python programs, in discussing future work, we’ve added discussion around more open-ended tasks like summarization where we might want to optimize a “soft” score (e.g., ROUGE [7]). In short, these problems are still approachable with Python but may require the language model to leverage another machine learning model.

Reviewer comment (Experiments):
What is the performance with different number of hypotheses? Specifically, in Table 1, it is important to know the performance with fewer number of initial generated hypotheses, such as 8. Comparing hypothesis summarization with directly generating 8 initial hypotheses can validate the importance of the hypothesis summarization stage.

Author response:
> What is the performance with different number of hypotheses? 

Thanks for this excellent question! We’ve performed additional experiments in Appendix B.1 to investigate the performance as the number of considered hypotheses changes and found that the accuracy of models increases consistently with more hypotheses from GPT-4.

Reviewer comment (Experiments):
In Table 1, the comparison of sample size and token size among different methods is unclear. Specifically, for hypothesis summarization, it is better to uniformly require the model to generate 8 programs for each of the 8 hypotheses for all problems, instead of only applying to 21 tasks, so that the sampling size is more comparable to the program prompting. Similarly, for human-selected hypotheses, it is unclear how many hypotheses are kept after filtering. It is better to always keep 8 hypotheses after filtering. In addition, it is unclear why the number of execution rounds varies for different methods. It is better to unify the setup for a fair comparison.

Author response:
> In Table 1, the comparison of sample size and token size among different methods is unclear. Specifically, for hypothesis summarization, it is better to uniformly require the model to generate 8 programs for each of the 8 hypotheses for all problems, instead of only applying to 21 tasks, so that the sampling size is more comparable to the program prompting. 

For hypothesis summarization, we agree that it is better and more rigorous to evaluate on all the tasks instead of a subset. Originally, a subset was used due to cost constraints, but we have now been able to access additional resources. We are currently running this experiment on the full 100 tasks and expect to have results before the end of the discussion period – please note, however, that we do not expect the final results on hypothesis summarization to meaningfully change.

> Similarly, for human-selected hypotheses, it is unclear how many hypotheses are kept after filtering. It is better to always keep 8 hypotheses after filtering. 

We will only keep the first correct hypothesis found for each task. So we are testing one hypothesis for each task in the human-selected experiment. We agree that selecting 8 hypotheses after filtering would be better for consistency; there are likely other clever oracles that one might use for filtering for this (e.g., using GPT-4 to find the most similar hypotheses to the ground truth), and we believe this would be a valuable exploration for future work.

> In addition, it is unclear why the number of execution rounds varies for different methods. It is better to unify the setup for a fair comparison.

Great point! We have unified all of the experiments to have three feedback rounds.

Reviewer comment (Evaluation):
The findings on SyGuS are divergent from the main evaluation, as the best result is achieved with purely code generation.

Author response:
> The findings on SyGuS are divergent from the main evaluation, as the best result is achieved with purely code generation.

Thank you for this point! We’d like to clarify that we don’t consider the SyGuS results as contradicting the overall result: indeed, our pipelines (both Program Only and Full) both perform significantly better than prior work. However, we do find that the natural language hypotheses are not a useful additional abstraction in this dataset. This is likely because the SyGuS problems are relatively easy.

We have also introduced a new dataset to demonstrate the usefulness of natural language hypotheses on non-ARC tasks. Specifically, we now investigate the cognitive-science-inspired inductive reasoning dataset, List Functions, from “The Child as Hacker” [1] which is fairly different from any of the current three datasets. As on the ARC datasets, this shows a clear advantage with the natural language hypotheses. 

| Method           | 64 hypotheses   |
|------------------|-----------------|
| Direct Prompting | 31%             |
| Program Only     | 59%             |
| Full             | 69%             |

[1] “The Child as a Hacker,” Rule et al. 2020

Reviewer comment (Evaluation):
Please provide a quantitative analysis on the failure mode; i.e., the percentage of error cases where none of the hypothesis is correct, and the percentage of error cases caused by the wrong generated programs.

Author response:
> Please provide a quantitative analysis on the failure mode; i.e., the percentage of error cases where none of the hypothesis is correct, and the percentage of error cases caused by the wrong generated programs.

Great suggestion! For the human-selected evaluation on ARC, we’ve now added a failure case analysis - namely, we were curious about the following: for what subset of the questions did the model fail because it failed to generate a correct hypothesis, and for what subset did it fail because it failed to implement a correct hypothesis? We looked at the 100 sampled tasks and the human-selected correct generated hypotheses (though, we reiterate here that “correct” is both subjective and somewhat arbitrary without experiments in line with those conducted in LARC). For 49% of them, at least one correct hypothesis was identified as correct. Of those selected, 33 (67%) led to correct programs. However, when given all correct language (i.e., the Human-Written Hypothesis evaluation), the model successfully implements 45% correctly. Of these, only one task is solved by the human-selected pipeline that isn’t solved by the human-written pipeline. This may suggest that there is a relationship between the tasks that the model can propose a hypothesis for and the tasks for which it can implement that hypothesis.

Reviewer comment (Experiments):
Please provide the full prompt including the few-shot demonstrations. The appendix only contains the zero-shot prompt. What is the performance of zero-shot prompting? How much does adding 1 or 2 problems in the prompt affects the performance?

Author response:
> Please provide the full prompt including the few-shot demonstrations. The appendix only contains the zero-shot prompt. What is the performance of zero-shot prompting? How much does adding 1 or 2 problems in the prompt affects the performance?

Thanks for the suggestion. We have included the full prompt for ARC and 1D-ARC in Figure A.8 and Figure A.9, including the two demonstrations. We test zero-shot prompting on the 1D-ARC dataset, keeping the number of generated hypotheses and programs per hypothesis the same. This gives us an accuracy of 71.3%, which is slightly lower than the performance with 2-shot prompting (73.1%). This shows that examples are helpful but not crucial in hypothesis generation. Note that we do not use demonstrations for the SyGuS and List Functions datasets.

Reviewer comment (Experiments):
The evaluation sets of ARC and 1D-ARC are too small. It is better to include at least 100 tasks.

Author response:
> The evaluation sets of ARC and 1D-ARC are too small. It is better to include at least 100 tasks.

Thank you for this excellent point! Originally, scaling this up was prohibitively expensive. However, we have now been able to access additional resources, allowing us to extend these results to 100 tasks. We refer the reviewer to the main paper for more details. All the conclusions remain consistent with the previous results, which demonstrate the effectiveness of our proposed pipeline.

Reviewer comment (Experiments):
While the method of generating and implementing hypotheses as Python programs is innovative, it may pose scalability challenges. For instance, generating a large number of hypotheses for complex problems could be computationally intensive and time-consuming. Moreover, the filtering process—whether automated or human-assisted—might not efficiently narrow down to the most effective hypotheses. To improve, the authors could explore more sophisticated algorithms for hypothesis generation that prioritize efficiency and scalability, possibly through more advanced heuristics or machine learning techniques.

Author response:
This is an excellent point - indeed, we emphasize the high cost of this approach throughout the paper, as well as some strategies that we took to attempt to mitigate it (e.g., since the primary cost comes from implementing rather than generating hypotheses, our summarization approach substantially reduces the necessary costs). We highlight multiple approaches to improve the scalability by focusing on selecting or extracting a smaller set of hypotheses with the language model; on the other hand, our method can also provide a flexible trade-off between performance and cost by varying the number of hypotheses and programs to search for. This is unachievable by the baseline method that directly prompts the answer. 
Recent work makes the reduced performance from the current search-space-reduction approaches less surprising, emphasizing that even advanced language models struggle to evaluate their own generations without external feedback [1].
Furthermore, we would like to highlight the flexibility of Python code from a program synthesis perspective. Before LLMs, program synthesis typically required operating over a small DSL with a very restrictive grammar, e.g. regular expressions, string manipulations, or 3D graphics [2,3,4,5]. However, we agree that this is an important point. To emphasize it, we have added a section on limitations and future work, where we discuss this.

Reviewer comment (Experiments):
The paper demonstrates success in specific datasets like ARC, 1D-ARC, and SyGuS, but it's unclear how well this method generalizes to other types of inductive reasoning tasks, particularly those with differing structures or complexity levels, or even cannot be solved with python programs. Also, the baselines are limited only with direct prompting and ablated baselines, with no baselines from related works. In other words, the range of experimental tasks presented is somewhat limited, potentially restricting the scope of the paper’s conclusions.

Author response:
Thanks for this point! In response, we’ve added a new cognitive-science-inspired inductive reasoning dataset, List Functions, from “The Child as Hacker” [6] which is fairly different from any of the current three datasets. As on the ARC datasets, this shows a clear advantage with the natural language hypotheses.

| Method           | 64 hypotheses   |
|------------------|-----------------|
| Direct Prompting | 31%             |
| Program Only     | 59%             |
| Full             | 69%             |

We also agree that there are always more datasets that we could include. We’d like to note that datasets like 1D-ARC are designed to vary the complexity and structure of ARC, and SyGus is a very differently structured task. In terms of tasks that cannot be solved by Python programs, in discussing future work, we’ve added discussion around more open-ended tasks like summarization where we might want to optimize a “soft” score (e.g., ROUGE [7]). In short, these problems are still approachable with Python but may require the language model to leverage another machine learning model. 

Thanks for raising this point. One of the reasons that we were excited to work on this direction is that there is not an extensive body of work that has explored inductive reasoning with language models. We will highlight that certain ablations can indeed be seen as corresponding to applying related works to the inductive reasoning task. For example, the program-only baseline without revisions can be seen as somewhat analogous to the “Program of Thoughts” prompting if applied to the inductive reasoning domain [8]. We’ve also added a language-only baseline that can be seen as analogous to a “Chain of Thoughts” baseline when applied to inductive reasoning [9]. We have also now modified our main text to incorporate this. We’ve also run the state-of-the-art DSL-based approach for ARC on our 100-task subset, finding that it performs slightly below the human-written score, solving 43 of the problems correctly. We’ve added this context in Section 3.2.

Reviewer comment (Theory):
The hypothesis proposal and selection process is essentially a search problem. The proposed iterative sampling and verification process is costly and inefficient from the perspective of search. The authors could consider more advanced search methods, such as DFS/BFS/MSTC, etc. Get some inspiration from the recent tree search prompting literature, like Tree-of-thoughts, reasoning-via-planning, etc.

Author response:
First, one should not consider our approach as competing with alternative search algorithms - indeed, they are complementary. One can actually interpret the iterative revision process as BFS: for each hypothesis implementation node, we expand it, revising each one after all others in the same “depth” have been revised. Unfortunately, other techniques like “Tree-of-thoughts” require the model to be able to assess the quality of a node, which we observed as one of their limitations. It is possible that one could implement an MCTS-inspired approach with access to a trainable language model acting as a reward model, in order to improve its ability to assess quality, but this would be a significantly different (albeit valuable) future work.

Reviewer comment (Theory):
How might this method perform tasks that are inherently difficult or perhaps impossible to encapsulate within a programmable framework?

Author response:
In principle, this framework should be applicable for virtually any inductive reasoning task that can be represented computationally — in practice, many more complex visual inductive reasoning tasks will require extending this implementation, such as using a compositional visual programming language to represent the program (e.g., VISPROG [10]). Moreover, if a task is truly unprogrammable, then we lose a core value of this work, namely the inductive bias and generalization that comes with a programmatic representation. Some challenges may also arise for models with a higher false-positive rate, which we’ve now noted in the limitations.

Reviewer comment (Reproducibility):
Could the authors clarify the missing elements in the appendix that might be pertinent to the paper's methodology or findings?

Author response:
Could you please elaborate on what elements you believe are missing? Happy to add any details!

Reviewer comment (Experiments):
Regarding the ARC tasks, what is the average duration, and why do most exceed the 4096 token limit imposed by many LLMs?

Author response:
We’ve clarified that the primary models we use (in the current experiments, gpt-4-0613) has a context window of 8k tokens. The average size of grids in ARC is (averaged over the 100 tasks we consider) is (10x10). In our experiment, there is only one task in ARC whose prompt exceed the context length of GPT-4, and 14 tasks are longer than the context of GPT-3.5, which is 4,096 tokens.

Reviewer comment (Experiments):
The Direct Prompting baseline, is it just few-shot prompting or Chain-of-thought prompting?

Author response:
The direct prompting baseline does not require the model to generate intermediate language. We’ve now also added a language-only baseline that can be seen as analogous to a “Chain of Thoughts” baseline when applied to inductive reasoning [5]. We find that, when not generating programs, regardless of whether the model generates the intermediate hypotheses or whether we provide human-written hypotheses, the pass rate is only marginally higher (from 17% without any language to 19% in either language-only case).

Reviewer comment (Novelty):
Which part of the pipeline is novel is not quite clear from the paper writing. The paper introduces every part of the proposed pipeline in intensive detail, but it is not quite clear which part of the pipeline is novel. Compared to earlier works like program-of-thoughts, the novel part appears to be generating natural language hypotheses before program generation and including a verification step to verify the correctness of the hypotheses. It is suggested to add a paragraph in the introduction to highlight which parts are novel and the contributions of the work.

Author response:
Thank you for this great point! We’ve added a short list of contributions to the introduction.

Reviewer comment (Presentation):
Some experiments, such as comparing the performance of GPT-3.5 and GPT-4, do not seem relevant to the main contribution of the paper. The results of these experiments could be moved to the appendix to avoid distraction.

Author response:
Thanks again for the suggestion - we’ve now moved the GPT-3.5 experiments to the appendix.

Reviewer comment (Presentation):
In Table 3, the names of the methods are different from those in Table 1. It is unclear whether "Full" in Table 3 corresponds to any method in Table 1.

Author response:
This is a great question. Because the problem sizes (in terms of the number of tokens necessary) and the tasks are easier for 1D-ARC, for the “Full” method, we evaluate all of the generated hypotheses. In contrast, a “Full” evaluation of all of the generated hypotheses for ARC would be prohibitively infeasible for us at the current cost of GPT-4. We’ve now clarified this in the text of the corresponding subsection. For the direct prompting method, we report the results from [2] for a direct comparison.

Reviewer comment (Presentation):
For the negative results presented in Sec. 3.4, it is suggested to summarize them in a table as well.

Author response:
Thank you for this suggestion! We’ve added a table to Sec. 3.4, but we’d like to clarify that we don’t consider the SyGuS results as negative. Indeed, both the Program-Only and Full hypothesis search pipelines perform significantly better than prior work. However, we do find that the natural language hypotheses are not a useful additional abstraction in this dataset. Likely because the SyGuS tasks are simply easier. We also introduce a new dataset to demonstrate the usefulness of natural language hypotheses on non-ARC tasks. Specifically, we now investigate the cognitive-science-inspired inductive reasoning dataset, List Functions, from “The Child as Hacker” [1] which is fairly different from any of the current three datasets. As on the ARC datasets, this shows a clear advantage with the natural language hypotheses. 

| Method           | 64 hypotheses   |
|------------------|-----------------|
| Direct Prompting | 31%             |
| Program Only     | 59%             |
| Full             | 69%             |

  
[1] “The Child as a Hacker,” Rule et al. 2020  
[2] LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations. Xu et al. 2023