Paper: Unleashing the Creative Mind: Language Model As Hierarchical Policy For Improved Exploration on Challenging Problem Solving (ICLR-2024)

Reviewer comment (Experiments):
Need ablation experiments to prove that the proposed tournament-based method is better than simple voting.

Author response:
> When we select the final answer among the groups of reasoning chains explored by our hierarchical policy, is our tournament-based selection approach better than direct majority voting?

We have added an experiment in our revised paper. We demonstrate that our tournament-based reasoning chain selection approach yields better final-answer accuracy over majority voting. Please see **Common Response** for more details.

Reviewer comment (Experiments):
For mathematical problems, the current some work has used code interpreter to greatly improve the results, such as (53.9% → 84.3%)[1], can the method in this paper be effective for this setting? From this perspective, the improvement brought by directly using LLM to output results in the paper is not significant, and code interpreter may be the key to solving math problems.

Author response:
> Can our approach be effective under the code-interpreter setting?

Thanks for your suggestion and we have cited the paper. We’d like to note that the focus of our paper is to introduce a visionary high-level leader that provides helpful hints to guide the low-level follower during problem solving. On the other hand, the code-interpreter line of works focus on integrating code interpreters into the detailed problem solving processes (which is the role of the low-level follower under our hierarchical policy framework), and these works have not delved into high-level strategies or thinking processes to our best knowledge. Therefore, our paper's focus is orthogonal to using code interpreter to improve challenging problem solving. In particular, code interpreter-enhanced low-level followers can be naturally integrated into our hierarchical policy framework.

To our best knowledge, the API for GPT-4 code interpreter was just released publicly last week for `gpt-4-1106-preview` [(see link here)](https://platform.openai.com/docs/assistants/tools/code-interpreter). We are actively working on implementations and experiments. We will try our best to add the results by the end of the rebuttal deadline. If we do not finish by then, we will add the results in final version.

**Update Nov. 22**

> Can our approach be effective under the code-interpreter setting?

We have obtained the results where `gpt-4-1106-preview` with code interpreter serves as the low-level follower policy in our approach along with the language model in the CoT + Sampling baseline. We adopt $n=4$ groups, each sampling $m=4$ reasoning chains. Results on MATH Level-5 are shown below. **We find that our hierarchical policy framework, combined with our tournament-based reasoning chain selection process, also yields better final answer accuracy under the code interpreter setting.**

|Baseline CoT + Sampling (GPT-4-1106-preview + code) | Ours-Retrieval (GPT-4-1106-preview + code) w/o tournament | Ours-Retrieval (GPT-4-1106-preview + code) w/ tournament |
|----|----|----|
|64.52 | 66.19 | **66.43** |



As a reference, we report the MATH Level-5 performance of different language models below, where each model samples a single CoT reasoning chain per problem (without sampling multiple reasoning chains or performing majority voting like in our previous settings). We find that utilizing code interpreter yields about 10\% final accuracy improvement for GPT-4 models, while it is not helpful for GPT-3.5.

| Base Model |GPT-3.5-0613 | GPT-4-0613 | GPT-4-1106-preview |
|----|----|----|----|
|w/o code interpreter |14.29|23.57|42.86|
|w/ code interpreter |12.14|33.57|52.14|

Reviewer comment (Experiments):
The evaluation datasets are not comprehensive enough. The authors only evaluate the approaches on a single dataset (the MATH dataset), and the relative evaluation size is small. Other math datasets (e.g., GSM8K, PRM800K) or other domain datasets in MMLU (e.g., Physics, Chemistry, etc) should be evaluated to demonstrate the generalization.

Author response:
> Evaluation over more datasets, like GSM8K and PRM800K.

Thanks for your suggestion! We have added additional experiments on GSM8K, and our approach continues to achieve better final-answer accuracy. Please refer to the **Common Response** for detailed results. 

(Note: The PRM800K also employs the MATH dataset, which is the same dataset our paper's experiments are based on.)

Reviewer comment (Experiments):
It is uncertain whether the improvement is from the hierarchical policy or the "self-evaluation" process when choosing the better reasoning chains. Previous research (e.g., "Language Models (Mostly) Know What They Know") suggests that LLMs like GPT-4 possess the ability to assess the likelihood that their output is correct.

Author response:
> Is our improvement coming from the hierarchical policy framework or the "self-evaluation" process when choosing the better reasoning chains?

Thanks a lot for your suggestion. We have added two ablation experiments in our revised paper. In the first experiment, we investigate whether the baseline CoT + Sampling performance can be improved through our tournament-based evaluation process. Results are shown in Table A below. We find that adding our tournament selection process to the baseline does not ourperform our approaches that adopt the hierarchical-policy framework, demonstrating that our hierarchical policy plays a significant role in enhancing LLM's ability to solve challenging problems.

In the second experiment, we investigate the effect of majority voting vs. our tournament selection on the final answer accuracy of our hierarchical policy approaches. Results are shown in Table B below. We find that for our hierarchical policy approaches, adopting our tournament selection process yields better final answer accuracy than using majority voting, so our tournament selection process is also helpful. Intuitively, this is because not all of the tactics and hints produced by our high-level leader are helpful, and some of them might mislead the low-level follower, potentially causing it to generate a consistent but wrong answer under a misleading high-level guidance. By evaluating reasoning chains using our tournament-based selection approach, we effectively remove those that exhibit reasoning mistakes and keep those that are more trustful.

**Table A: Effect of our tourament-based reasoning chain selection approach on the CoT + Sampling baseline. For the baseline, we randomly partition the 64 sampled reasoning chains per problem into $n=4$ groups, each having $m=16$ reasoning chains. We use GPT-3.5 as the language model for our low-level follower and the CoT Sampling Baseline.**
| Method      | Baseline CoT + Majority Voting | Baseline CoT + Tournament  | Ours - Tactic (w/ Tournament)  | Ours - Retrieval (w/ Tournament) |
|------------------|------|-------------|-------------|-------------|
| Answer Accuracy  | 22.14 | 22.86   |**25.00**  | **27.85**   |


**Table B: Effect of majority voting and our tournament-based reasoning chain selection on the final-answer accuracy of CoT + Sampling baseline and our hierarchical policy approaches. For "Majority Voting", we directly perform majority voting over the 64 sampled reasoning chains per problem. For "Majority Voting over Groups" and "Tournament", we adopt $n=4$ groups, each having $m=16$ reasoning chains. We use GPT-3.5 as the language model for our low-level follower and the CoT Sampling Baseline.**
| Method |  Baseline CoT | Ours - Tactic | Ours - Retrieval |
|------------------|------|-------------|-------------|
| Majority Voting | 22.14 | 21.79 | 23.57 |
| Majority Voting over Groups | 19.70 | 20.24 | 23.39 |
| Tournament | 22.86   |**25.00**  | **27.85** |

Reviewer comment (Evaluation):
What's the motivation of the "Grouped-Majority Recall" metric? A more intuitive idea may be the percentage of questions whose ground truth answer exists in at least one of the answers.

Author response:
> What's the motivation of the "Grouped-Majority Recall" metric?

The motivation of our Grouped-Majority Recall metric is presented in the second and the fourth paragraph of Sec. 4.1, along with Figure 4. We have also made our motivation more clear in our revised paper.

In detail, the purpose of our metric is to assess the **“visibility”** of the correct answers among solutions generated along the exploration process. A correct answer is “visible” if it not only exists in at least one of reasoning chains, but also occupies a substantial proportion of them, even though it might not be the majority answer. *We find that standard metrics like accuracy and recall do not perfectly align with our goal.* This is because standard recall poorly correlates the prominence of correct answers, and standard accuracy does not reflect the scenario where LLM identifies correct solutions by a reasonable chance (and more than once), but the correct solutions are submerged during majority voting. On the other hand, our Grouped-Majority Recall metric allows correct answers that are identified by a reasonable chance but become obscured in majority voting to emerge, while ensuring that all correct answers recognized by our metric take up the majority in at least one reasoning group and are represented in more than one reasoning chains.

> A more intuitive idea may be the percentage of questions whose ground truth answer exists in at least one of the answers.

The percentage of questions whose ground truth answer exists in at least one of the answers is the "standard recall" metric referred in the last comment. From our analysis, we find that the standard recall metric is not an ideal metric to assess the visibility of correct answers, because as the number of reasoning chain samples increases, the standard recall steadily improves, but the standard accuracy quickly plateaus after a few reasoning chain samples (as shown in Fig. 4a of the paper). Thus, the standard recall metric does not correlate well with the prominence of correct answers.

Reviewer comment (Experiments):
In the "tournament-based approach", GPT-4 is used to select the better reasoning chain as the final solution. Because of the "self-evaluation" ability of LLMs, have you tried to use the majority vote strategy to obtain the final answer as an ablation experiment and compute the accuracy? In GPT-3.5 based approaches, is the "tournament" based on GPT-4 or GPT-3.5 (you mentioned that the GPT-4 is prompted to compare the current chains with (i+1)-th chain (Section 3) )?

Author response:
In the second experiment, we investigate the effect of majority voting vs. our tournament selection on the final answer accuracy of our hierarchical policy approaches. Results are shown in Table B below. We find that for our hierarchical policy approaches, adopting our tournament selection process yields better final answer accuracy than using majority voting, so our tournament selection process is also helpful. Intuitively, this is because not all of the tactics and hints produced by our high-level leader are helpful, and some of them might mislead the low-level follower, potentially causing it to generate a consistent but wrong answer under a misleading high-level guidance. By evaluating reasoning chains using our tournament-based selection approach, we effectively remove those that exhibit reasoning mistakes and keep those that are more trustful.

**Table B: Effect of majority voting and our tournament-based reasoning chain selection on the final-answer accuracy of CoT + Sampling baseline and our hierarchical policy approaches. For "Majority Voting", we directly perform majority voting over the 64 sampled reasoning chains per problem. For "Majority Voting over Groups" and "Tournament", we adopt $n=4$ groups, each having $m=16$ reasoning chains. We use GPT-3.5 as the language model for our low-level follower and the CoT Sampling Baseline.**
| Method |  Baseline CoT | Ours - Tactic | Ours - Retrieval |
|------------------|------|-------------|-------------|
| Majority Voting | 22.14 | 21.79 | 23.57 |
| Majority Voting over Groups | 19.70 | 20.24 | 23.39 |
| Tournament | 22.86   |**25.00**  | **27.85** |

> In GPT-3.5 based approaches, is the "tournament" based on GPT-4 or GPT-3.5 (you mentioned that the GPT-4 is prompted to compare the current chains with (i+1)-th chain (Section 3) )?

In our tournament-based reasoning chain selection process, we use GPT-4 to compare the current reasoning chain with the $(i+1)$-th chain, where the reasoning chains can be generated by either GPT-3.5 or GPT-4.

We further add an ablation study in Table 5 of the revised paper to compare using GPT-3.5 or GPT-4 to conduct our tournament-based reasoning chain selection process. We observe that GPT-4 demonstrates stronger performance as a reasoning chain selector than GPT-3.5 with a limited number of comparison repetitions (e.g., $k = 1$). Additionally, with only $k=1$ comparison repetition, GPT-4 already leads to a noteworthy enhancement of final answer accuracy over the CoT Sampling baseline.

Reviewer comment (Theory):
Some descriptions of the method/experiment are confusing. Equation (1) and the relevant text is an example. The authors integrate w.r.t. $h$, so they treat $Pr(A|h,Q)$ as a probability density function so the integral $Pr(A|Q)$ should be a probability mass function. Yet the authors use the same notation $Pr$ which is quite confusing. More importantly, although we can understand what the authors would like to express after reading the whole section, the equation itself is invalid as $h$ is a discrete random variable rather than a continuous random variable so you cannot integrate w.r.t. $h$. Please note that mathematical notations in a paper is for helping readers to understand your idea more easily; but Equation (1) is not helping but instead making it even harder to understand. Actually, the key point of the section is just one sentence: "More generally, our strategy samples all the different hints returned by $\pi_{high}$ with equal probabilities.". And this is already clear enough.

Author response:
> Clarification of Equation 1

Thanks for your suggestion. The answer, hint, and question spaces are all discrete, so we have replaced the integral sign with the sum sign and changed our equation (1) to

$\mathrm{Pr}(A|Q) = \sum_{h}\mathrm{Pr}(A|h, Q) \cdot \text{Unif}(h\in H)\cdot \mathrm{Pr}(H|Q)$

in our updated paper.

(As a side note, we originally used the integral sign as we were using a more general measure theory-style notation, under which one could use the integral sign to integrate over any measurable space, such as discrete spaces with counting measure and $\mathbb{R}^n$ with Lebesgue measure. In our original Equation 1, the integral was with respect to the counting measure over our discrete hint space, i.e., $\mathrm{Pr}(A|Q) = \int_{h}\mathrm{Pr}(A|h, Q) \cdot \text{Unif}(h\in H)\cdot \mathrm{Pr}(H|Q) d\mu(h)$ and $\mu$ is the counting measure. In our original paper, we omitted $d\mu(h)$ for brevity, which possibly caused the confusion.)

> Core message of our "Probabilistic Interpretation" Section

Thanks for your suggestion. As suggested, we have highlighted the sentence `our strategy samples all the different hints returned by $\pi_{high}$ with equal probabilities` in our updated paper.

We'd like to also clarify that the purpose of our "Probabilistic Interpretation" section is to provide an intuition of why our proposed hierarchical policy approach endows LLMs with better exploration ability in solving challenging reasoning problems, *compared to previous approaches that directly sampling multiple reasoning chains to explore the solution space*. Our analysis reveals that directly sampling reasoning chains is prone to stucking in a single or a few high-probability tactics, while our hierarchical approach allows tactics to be more uniformly sampled, which is beneficial for solving challenging math problems that require out-of-the-box thinking.

Reviewer comment (Writing):
Similar feelings also appear in the section introducing the "Grouped-Majority Recall" metric, where the authors use quite long paragraphs to explain the details of it but the organization is not quite good and thus make it not easy to get the motivation of proposing this new metric. We generally suggest the authors to improve the expression and organization in these sections for better readability.

Author response:
> Improve the writing organization for our motivation of "Grouped-Majority Recall"

Thanks for your suggestion. We have revised our paper and added multiple boldface highlights to make the core messages clear when we describe the motivation of our Grouped-Majority Recall metric. 

In detail, the goal of our metric is to assess the **“visiblity”** of the correct answers among solutions generated along the exploration process. *A correct answer is “visible” if it not only exists in at least one of reasoning chains, but also occupies a substantial proportion of them, even though it might not be the majority answer.* The second paragraph of Sec. 4.1, along with Figure 3, demonstrates why standard metrics like accuracy and recall are not suitable for our goal. The third paragraph of Sec. 4.1 explains the definition of our Grouped-Majority Recall metric. The fourth paragraph of Sec. 4.1 demonstrates why our Grouped-Majority Recall metric better measures the visibility of correct answers compared to the standard accuracy and recall metrics.

Reviewer comment (Reproducibility):
For retrieval-based hints generation, after you find similar examples in the training data, how do you identity the hints for these examples? The original MATH dataset doesn't contain such annotations. Do you generate the hints by LLM? If so, what's the accuracy of these hints?

Author response:
> Clarification on retrieval-based hint generation

After the high-level leader retrieves a relevant problem along with its step-by-step solution, the problem and the solution directly serves as the hint for the low-level follower to carry out the problem-solving process (see Table 10 in Appendix for an illustration). There are no further prompting to generate (more concise and high-level) hints using the retrieved problem and solution.

Reviewer comment (Experiments):
Limited Dataset Representation: The study's findings are primarily based on the MATH dataset, which raises concerns about the model's performance on other datasets that present different challenges, such as the GSM8k. This limits the understanding of the model's adaptability and effectiveness across various types of reasoning tasks.

Author response:
> Evaluation over GSM8k

Thanks for your suggestion! We have added additional experiments on GSM8K, and our approach continues to achieve better final-answer accuracy. Please refer to the **Common Response** for detailed results.

Reviewer comment (Reproducibility):
Ambiguity in Model Details: The paper does not specify which versions of GPT-3.5 and GPT-4 were used, nor does it detail the hyper-parameters involved. Such information is critical for replicating the study.

Author response:
> Version of GPT-3.5/GPT-4 along with their hyperparameters

We use GPT-3.5-turbo-0613 and GPT-4-0613 for all the experiments. For hyperparameter details, we set the decoding temperature to 0.3 for tournament-based reasoning chain selection and 0.7 otherwise (i.e., for hint generation from the high-level leader, reasoning chain generation from the low-level follower, along with the baselines). These hyperparameters were briefly mentioned in the first paragraph of our experiment setup. For clarity purposes, we have added an additional section (Appendix B in the updated PDF) to describe the GPT-3.5/4 settings used in the paper.

Reviewer comment (Evaluation):
Cost Analysis Omission: There is no comprehensive analysis of the computational costs associated with different methods, including the number of tokens generated and encoded. Such an analysis is essential to evaluate the model's efficiency and practicality.

Author response:
> Cost analysis comparison.

Thanks for your suggestion. We compare the cost between our approach and the CoT + Sampling Baseline below (unit in dollars; tournament performed with GPT-4; a total of 64 reasoning chains are generated per question). We also add this table in Appendix Section C. While our approach is slightly more expensive than the CoT baseline when GPT-3.5 serves as the low-level follower, it is more cost efficient than the baseline when GPT-4 serves as the follower, and moreover yields better final answer accuracy.


| **Method** | CoT + Sampling Baseline | Ours - Tactics | Ours - Retrieval | CoT + Sampling Baseline | Ours - Tactics | Ours - Retrieval |
|-------------------|---------------|-------------------------------|---------------------|-------------|----------------------------|-------------------|
| **Low-Level Follower** | GPT-3.5 | GPT-3.5 | GPT-3.5 | GPT-4 | GPT-4 | GPT-4|
| **Sampling Cost**      | 10.72           | 11.71            | 9.83                | 204.16        | 191.11         | 188.47           |
| **Tournament Cost**    | None           | 2.42                          | 4.45                | None         | 4.52                       | 5.92             |
| **Total Cost**         | 10.72           | 14.12            | 14.28               | 204.16        | 195.63         | 194.40           |

Additionally, we also compare the average number of input and output tokens between the baseline and our approach. The number of output tokens is calculated over the sum of 64 reasoning chains sampled per question. (For both GPT-3.5 and GPT-4, input tokens cost half as much as output tokens.)

| **Method** | CoT + Sampling Baseline | Ours - Tactics | Ours - Retrieval | CoT + Sampling Baseline | Ours - Tactics | Ours - Retrieval |
|--------------------|-----------------|------------------|---------------------|---------------|----------------|-------------------|
| **Low-Level Follower** | GPT-3.5 | GPT-3.5 | GPT-3.5 | GPT-4 | GPT-4 | GPT-4|
| Avg. # Input Tokens Per Question (including tournament in our approach) | 0.11K           | 0.73K            | 1.88K               | 0.11K         | 0.88K          | 2.00K      |
| Avg. # Output Tokens Per Question (including tournament in our approach)| 38.20K          | 41.61K           | 34.32K              | 24.25K        | 22.85K         | 22.14K            |