Paper: Large Language Models as Analogical Reasoners (ICLR-2024)

Reviewer comment (Experiments):
The paper emphasizes the importance of instructing the LLM to generate *distinct* exemplars, but there is no ablation performed for this specific aspect of the method.

Author response:
Thank you for pointing this out. Following is the ablation result on instructing the LLM to generate *distinct* exemplars or not: 
- GSM8K non-distinct vs distinct: 75.9 vs 77.8
- MATH non-distinct vs distinct: 35.2 vs 37.3

Reviewer comment (Theory):
It seems unlikely that the generated exemplars are literally retrieved from memory in the sense that they are in human reasoning. It seems more likely that these are novel problems generated by the LLM, based on general statistical knowledge. I don't think this really undermines the usefulness of the approach, but it might be worthwhile to briefly discuss this issue.

Author response:
This is an excellent point and you are correct. We observed that some of the self-generated exemplars resemble the potential training data of the LLM (e.g., GSM8K training set, Geeksforgeeks coding problems), but many of the exemplars are newly generated and appear to be an interpolation of problems seen by the LLM that are related to the target problem. We find this phenomenon very intriguing, suggesting that LLMs may be capable of generating useful exemplars beyond the existent/seen exemplars.

Reviewer comment (Reproducibility):
The caption for Table 1 mentions that an in-context demonstration was used for the davinci models, but I couldn't find any explanation of this description (e.g., does this include an in-context training example, or merely a demonstration of the formatting?).

Author response:
Thank you for pointing this out. We added the description of in-context demonstration in Appendix D.2. It provides an example of formatting the generation of exemplars and solution to the initial problem.

Reviewer comment (Theory):
I am curious to hear the authors' thoughts regarding whether the generated exemplars are genuinely retrieved from memory, or are novel problems based on general statistical knowledge. The latter case seems more consistent with the memory mechanisms in LLMs (and with their tendency to generate fabricated but plausible sounding information), but it is not something that has been considered much in the psychology literature on analogical reasoning, and it is somewhat more difficult to understand how it could improve performance.

Author response:
This is an excellent point and you are correct. We observed that some of the self-generated exemplars resemble the potential training data of the LLM (e.g., GSM8K training set, Geeksforgeeks coding problems), but many of the exemplars are newly generated and appear to be an interpolation of problems seen by the LLM that are related to the target problem. We find this phenomenon very intriguing, suggesting that LLMs may be capable of generating useful exemplars beyond the existent/seen exemplars.

Reviewer comment (Novelty):
Limited technical contribution. Although I think this paper is a good example of how to perform prompting engineering when solving zero-shot mathematical reasoning and code generation tasks, it is still more like a trick to an existing method (analogous to CoT->zero-shot CoT, in this case is retrival few-shot -> self-generated few-shot).

Author response:
Thank you for your valuable feedback. We would like to highlight that our approach makes significant methodological contributions, beyond prompt engineering or the transition from retrieval few-shot to self-generated few-shot:
- **New LLM reasoning paradigm**: We introduce a new problem-solving paradigm for LLM, inspired by analogical reasoning in psychology. As Polya's book "How to Solve It" (2004) states, when solving a new problem, humans draw from how they solved related problems in the past. We are the first to show that this strategy aids LLMs in solving complex reasoning problems (Section 1).
- **Design of new techniques**: To ground this strategy in LLM prompting, we analyzed the design space and present effective techniques:
  - Diversity in generated exemplars (Section 4.1)
  - Sequential vs independent generation of exemplars (Section 4.1)
  - Generation of high-level knowledge in addition to low-level exemplars (Section 4.2)
- **Impact of generation v.s. retrieval**: Our method, based on generation rather than retrieval, offers greater generality and flexibility. It unlocks the generation of various forms of relevant information, such as exemplars, knowledge, formulas, tutorials, etc. (as experimented in Section 4.2). This goes beyond traditional retrieval methods that were confined to a pre-defined set (e.g., GSM8K train set), and empirically yields superior results too (Table 4).

Reviewer comment (Experiments):
The codeforces dataset only contains 50 questions; perhaps it is too small to make claims on the improvements (~2% is only one more solved question). Any experiments on larger code generation tasks, e.g., HumanEval?

Author response:
This is a great point. The reason we used Codeforces 2023 for evaluation, despite its smaller size, was to prevent test set contamination; larger code generation datasets such as HuanEval and older versions of Codeforces may be used for LLM training, introducing the risk of contamination. To mitigate the issue of the small evaluation set, we repeated the experiments multiple times, each time using different LLM output samples, and then reported the average results (Section B).

We also evaluated our method on the larger, older Codeforces dataset from AlphaCode, involving 300 coding problems. Our prompting method outperformed the baseline CoT prompting by 4% in this setting as well.

Reviewer comment (Evaluation):
There is error analysis for incorrectly solved tasks, how about correctly solved questions? How many generated exemplars are wrong, but the solution to the new question is correct (i.e., it is known that sometimes LLMs can few-shot generalize from wrong exemplars)?

Author response:
Thank you for pointing this out. You are right. While not too often, there are cases where generated exemplars are wrong but the solution to the new question is correct. This result aligns with the previous finding that LLMs can few-shot generalize from wrong exemplars. Below are the details of 50 correctly solved problems and 50 incorrectly solved problems from GSM8K+MATH:
- 50 correctly solved problems:
  - (15/50) Generated exemplars are incorrect
  - (35/50) Generated exemplars are correct
- 50 incorrectly solved problems:
  - (22/50) Generated exemplars are incorrect
  - (28/50) Generated exemplars are correct

Reviewer comment (Experiments):
Will the language model be easily distracted when the generated examples are irrelevant or incorrect? The experiments in section 6.6 should provide more details. For instance, it should specify how many problems in MATH fail due to incorrect and irrelevant generated examples.

Author response:
Great question. Below is the analysis of 50 correctly solved problems and 50 incorrectly solved problems from GSM8K+MATH. Overall, when the generated examples are irrelevant or incorrect, the LLM produces more wrong answers (22) than correct answers (15), but the gap is not large, indicating that LLM is not distracted to a critical extent.
- 50 correctly solved problems:
  - (6/50) Generated exemplars are irrelevant
  - (9/50) Generated exemplars are relevant but incorrect
  - (35/50) Generated exemplars are relevant and correct
- 50 incorrectly solved problems:
  - (10/50) Generated exemplars are irrelevant
  - (12/50) Generated exemplars are relevant but incorrect
  - (28/50) Generated exemplars are relevant and correct

Reviewer comment (Experiments):
Can this method be integrated with self-consistency decoding? For instance, could the majority voting result from multiple reasoning chains with the generated knowledge, lead to better outcomes?

Author response:
This is a great suggestion. Yes, our method can be integrated with self-consistency and this indeed improves the performance:
- GSM8K
  - Ours: 77.8
  - Ours + self-consistency: 85.3
- MATH  
  - Ours: 37.3
  - Ours + self-consistency: 46.0

In particular, we find that self-consistency reduces the failure case of self-generating irrelevant examplers and getting distracted. This suggests that self-consistency and our analogical prompting can effectively complement each other.

Reviewer comment (Experiments):
Can a verification stage be added to filter out incorrect or irrelevant examples to further improve performance?

Author response:
This is a very interesting idea. We experimented with this idea by sampling multiple exemplars, filtering out incorrect or irrelevant exemplars using GPT-4 as the verifier (i.e. prompting GPT-4), and then re-prompting the original LLM with the remaining exemplars to solve the target problem. We found this improves the task performance (e.g. 2% improvement in the MATH task).

Reviewer comment (Novelty):
It would be nice to conceptually compare the work against the neural symbolic method [1].  

[1] Zhang, Hanlin, et al. "Improved logical reasoning of language models via differentiable symbolic programming." arXiv preprint arXiv:2305.03742 (2023).

Author response:
Thank you for pointing us to the related work. We have added a citation to the neural symbolic work in our paper. Our approach, analogical prompting, is distinct yet complementary to neural symbolic methods. We present a high-level LLM reasoning strategy, demonstrating how generating related exemplars through analogy aids in problem-solving. Neural symbolic methods could further complement the problem-solving stage, which is an interesting avenue for future research.