Paper: THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR-2024)

Reviewer comment (Evaluation):
Since TP outperforms ToT by a surprising and counterintuitive amount, it is critical to compare the two methods in terms of token cost, which is a controllable quantity for both TP and ToT. The authors acknowledge the importance of comparing token cost in their discussion of Figure 5: “1-layer TP outperforms ToT by a large margin in different LLM backends but shares similar token expenses.” However, this figure applies only to the shortest-path task.

Author response:
Q3: The Computation cost analysis on Creative writing and LLM-agent Planning tasks.

A3: Thanks for this valuable comments! We agree that TP is more computationally expensive than ToT as TP needs to explore a set of analogous problems to update the initial input problem-solving. ToT uses 790.1K tokens for Creative writing while TP uses 2869.1K tokens. For the LLM-agent Planning task, it is challenging to accurately calculate the number of tokens due to some dependency issues of ALFWORLD. However, we managed to roughly estimate that Reflexion costs around 300 USD to run 6 trails while the different variant models of TP cost 300-400 USD to run 6 trails based on the billing information from an LLM research institute/company. 

To address the concerns that TP achieves better performances than baselines by simply using more tokens, we rerun CoT 10 times and max-vote the best results (CoT-SC [5]). Please refer to A2 in response to Reviewer 39VY for detailed experiment results. Our finding is that 1-layer TP+IO already outperforms CoT-SC. This indicates the efficacy of the proposed TP results from the novel analogical reasoning approach rather than only increasing the token cost. We will aim to further improve the efficacy of the proposed TP while reducing its computation cost in our future work. One solution is to incorporate the task-related knowledge in \texttt{LLM Propose} module. Thus, TP could propose the analogous problems that most help the input problem-solving. And we will add the above analysis to the Experiment part of the final version.

Reviewer comment (Experiments):
The results of the shortest-path evaluation are likely to depend very sensitively on how the graph definition is represented in natural language for insertion into the prompt, and small changes in representation could cause large performance differences, potentially favoring certain prompt approaches over others.

Author response:
Q2: The impact of how graphs are encoded into prompts in natural languages may affect the performances of different methods.

A2: Thanks for this valuable comment and we have added a detailed analysis in Appendix H.2! In our work, we initially adopted the straightforward manner to encode input graphs into the collection of node lists, edge lists, and edge distance lists. We denote such graph encoding method as $\textbf{Adjacency}$.

Previous works have shown that changing the prompt exemplars can impact the reasoning performances of prompting methods. As there are multiple ways to encode the input graphs into sequential data, we further consider two additional graph encoding methods including $\textbf{Edge Description}$ [1] and $\textbf{Graph Modeling Language}$ [2,3]. 

For an input graph with $N$ nodes, Edge Description represents the input graph as "In an undirected graph, the nodes are numbered from 0 to $N$, and the edges are represented as: an edge between node $i$ and node $j$ with distance $LENGTH$,$\cdots$.". Graph Modeling Language converts the input graph into the following sequence "graph[comment "This is an undirected graph." node [id 0] $\cdots$ node [id $N$] edge [label "Edge between node $i$ and node $j$ with distance $LENGTH$"] $\cdots$.]"

As shown in Table 1 below, different graph encoding methods indeed impact the performances of different prompting approaches. However, the proposed TP still enjoys an over 10\% absolute increase in finding the optimal shortest path when adopting Edge Description and Graph Modeling Language to encode graphs into sequences. $\textbf{This indicates the robustness of TP against different graph encoding methods.}$

Notice that we do not aim to propose the best way for LLMs to understand graphs. Instead, we aim to propose a general analogical approach to enhance the complex reasoning ability of LLMs across different tasks. As discussed in Section 5, TP achieves significant performance gains in different tasks not limited to Shortest-path reasoning.

Reviewer comment (Reproducibility):
No token costs are provided for the other two experiments, making it hard to validate the paper’s central finding.

Author response:
Q3: The Computation cost analysis on Creative writing and LLM-agent Planning tasks.

A3: Thanks for this valuable comments! We agree that TP is more computationally expensive than ToT as TP needs to explore a set of analogous problems to update the initial input problem-solving. ToT uses 790.1K tokens for Creative writing while TP uses 2869.1K tokens. For the LLM-agent Planning task, it is challenging to accurately calculate the number of tokens due to some dependency issues of ALFWORLD. However, we managed to roughly estimate that Reflexion costs around 300 USD to run 6 trails while the different variant models of TP cost 300-400 USD to run 6 trails based on the billing information from an LLM research institute/company. 

To address the concerns that TP achieves better performances than baselines by simply using more tokens, we rerun CoT 10 times and max-vote the best results (CoT-SC [5]). Please refer to A2 in response to Reviewer 39VY for detailed experiment results. Our finding is that 1-layer TP+IO already outperforms CoT-SC. This indicates the efficacy of the proposed TP results from the novel analogical reasoning approach rather than only increasing the token cost. We will aim to further improve the efficacy of the proposed TP while reducing its computation cost in our future work. One solution is to incorporate the task-related knowledge in \texttt{LLM Propose} module. Thus, TP could propose the analogous problems that most help the input problem-solving. And we will add the above analysis to the Experiment part of the final version.

Reviewer comment (Writing):
Some typos: in page 9, the last sentence of section 6, the quotation marks should be `` ''.

Author response:
We really appreciate your suggestions and we have corrected the quotation marks in the last sentence of section 6 on page 9.

Reviewer comment (Experiments):
Can we combine TP with CoT to achieve further performance improvements?

Author response:
Thanks very much for the valuable suggestion! And we have added an additional section in Appendix H.1 for detailed analysis. We adopt CoT to instantiate the $\texttt{LLM Solve}$ module of TP and test the obtained model, namely TP+CoT, on the Shortest-path Reasoning Task. In addition, we implement one and two layers of TP propagation for TP+CoT, leading to two variant models: TP (1-layer)+CoT and TP (2-layer)+CoT. We rerun CoT with GPT-3.5-turbo to ensure a fair comparison with two variant models due to the changing behavior of the GPT model [1]. Thus, the obtained performance of CoT is slightly different from that in our manuscript. We use the 3 metrics, namely Optimal Rate (OR↑), Feasible Rate (FR↑), and Over-Length Rate (OLR↓), as introduced in Section 5.1 for evaluation. We run all the models under 0-shot, 1-shot, and 5-shot settings.

[Table 1: Comparison between IO, CoT, CoT-SC, TP+IO, and TP+CoT in the Shortest-path Reasoning task.]
| Method            | 0-shot OR↑ | 0-shot FR↑ | 0-shot OLR↓ | 1-shot OR↑ | 1-shot FR↑ | 1-shot OLR↓ | 5-shot OR↑ | 5-shot FR↑ | 5-shot OLR↓ |
|-------------------|------------|------------|-------------|------------|------------|-------------|------------|------------|-------------|
| IO                | 0.33       | 0.50       | 0.17        | 0.62       | 0.86       | 0.15        | 0.61       | 0.90       | 0.27        |
| TP (1 Layer)+IO   | 0.60       | 0.82       | 0.14        | 0.71       | **0.89**   | 0.09        | 0.70       | **0.91**   | 0.14        |
| TP (2 Layer)+IO   | **0.65**   | **0.89**   | **0.12**    | **0.74**   | **0.89**   | **0.07**    | **0.73**   | **0.91**   | **0.10**    |
| CoT               | 0.30       | 0.42       | **0.14**    | 0.62       | 0.87       | 0.15        | 0.63       | 0.97       | 0.20        |
|CoT-SC(n=10) | 0.46	|0.55	|0.17	|0.67	|0.90	|0.13	|0.66	|0.97	|0.18
| TP (1 Layer)+CoT  | 0.53       | 0.75       | 0.19        | 0.68       | **0.91**   | 0.11        | 0.72       | **0.98**   | 0.15        |
| TP (2 Layer)+CoT  | **0.60**   | **0.83**   | 0.18        | **0.71**   | 0.89       | **0.09**    | **0.73**   | 0.96       | **0.13**    |


As shown in Table 1 above, TP (1 Layer)+CoT and TP (2 Layer)+CoT outperform CoT with significant performance gains, especially under the 0-shot setting. This indicates that TP enjoys plug-and-play enhancement for different prompting methods. Moreover, we find CoT and TP+CoT could not beat their IO prompting counterparts in the Shortest-path Reasoning task under 0-shot and few-shot settings, which is consistent with our findings and analysis in Section 5.1.

Reviewer comment (Experiments):
The authors are encouraged to compare TP with Self-refined LLM reasoning methods [1,2] since TP also manages to refine the solution to the input problems in LLM Aggregation module.

Author response:
We thank the reviewer for bringing our attention to these works. We have added an additional section in Appendix G (Pages 25-26) to compare the proposed work (TP) with self-refined LLM reasoning in detail and highlight our contributions. The self-refined LLM reasoning, also known as self-critic, is an emerging technique to rectify the mistakes and hallucinations during the inference time of LLM reasoning [1,2]. Recent works in self-refined LLM reasoning guide LLMs to examine the correctness of their output with internal [3] or external feedback [4], and further refine their output if needed.  

The $\texttt{LLM Aggregate}$ module of TP shares similar intuition with self-refined LLM reasoning methods, since it can yield strategies for improving the input problem-solving with the insights of solving $\textbf{analogous}$ problems. By analogical reasoning, $\texttt{LLM Aggregate}$ can assess a diverse set of strategies and choose the best one for refinement based on the connection between the input problem and its analogous counterparts. Instead, existing self-refined LLM reasoning methods cannot utilize the knowledge of solving analogous problems to refine the input problems as they cannot prompt LLMs to perform analogical reasoning. We will explore the application of TP in self-refined LLM reasoning in the future.

Reviewer comment (Evaluation):
In shortest path reasoning tasks, does TP sometimes deteriorate the solutions to some testing instances instead of improving them?

Author response:
Thank you very much for this question! We further divide the cases where TP changes the initial solutions to input problems by analogical reasoning into 2 categories: 

- Improved: the percentage of cases where TP changes the invalid solutions to valid ones and improves the valid solutions to better ones, 

- Deteriorated: the percentage of cases where TP changes the valid solutions into invalid ones and changes the valid solutions to worse (but also valid) ones.

We compute the probability of these two categories under 0-shot, 1-shot, and 5-shot in the Shortest-path Reasoning task. The results are in the following table.

|                |          |      0-shot       |             |          |       1-shot      |             |          |       5-shot      |             |
|----------------|---------------|-------------|-------------|---------------|-------------|-------------|---------------|-------------|-------------|
| Backends       | Palm-2        | GPT-3.5     | GPT-4       | Palm-2        | GPT-3.5     | GPT-4       | Palm-2        | GPT-3.5     | GPT-4       |
| Improved       | 68.97%         | 85.71%      | 86.96%      | 63.77%         | 83.72%      | 73.98%      | 71.74%         | 88.95%      | 100%        |
| Deteriorated   | 31.03%         | 14.29%      | 13.04%      | 36.23%         | 16.28%      | 26.02%      | 28.26%         | 11.05%      | 0%          |

As shown in the table above, TP improves more initial solutions in the Shortest-path Reasoning task rather than deteriorates them. This indicates that TP can greatly refine the initial solutions, which are obtained by reasoning from scratch, to enhance the complex reasoning ability of LLMs by analogical reasoning. Moreover, we observe that TP is more capable of improving initial solutions when deployed on more powerful LLMs such as GPT series. In our future work, we aim to further improve the efficacy of the TP framework and delve into the theoretical analysis behind analogical reasoning with LLMs.