Paper: Alphazero-like Tree-Search can guide large language model decoding and training (ICLR-2024)

Reviewer comment (Evaluation):
The performance improvement is unclear and somewhat unconvincing.  
- **GSM8k:** The results are inferior compared to the previous work (ToT-BFS).  
- **Game24:** Despite MCTS-$\alpha$ winning, it requires **nearly twice** the number of tokens compared to other approaches.  
- **PrOntoQA:** Similarly, it requires **nearly twice** the number of tokens compared to other approaches.  
- **RLHF:** (See the next point)

Author response:
Thanks for your comment. We totally agree with your observation. There are three things we want to clarify:
1. **We want to clarify that TS-LLM’s DFS/BFS variants are different from that in Tree of Thought (ToT). They are TS-LLM’s variants instead of ToT baselines.** The main difference is that we use a learned value function instead of prompting advanced LLMs, which is one of our important contributions. We have shown in Table 4 the result of BFS and MCTS-alpha on Game24 by prompting LLaMA2-7B model as value function and ORM. It can exactly be regarded as an instance of ToT over small LLMs. Our conclusion is that prompting a small LLM can not offer accurate value estimation and the performance of tree-search is even worse than simple CoT baseline.
2. For path@1 results in Table 2, MCTS-alpha and MCTS-rollout do consume more tokens and it is reasonable to question whether it is mainly because of more token consumptions. Our Q4’s experiment can help to clarify such an issue. We additionally conduct experiments on search aggregation shown in Figure 3, where you can scale up your token consumption by running multiple times of search in MCTS/MCTS-alpha/MCTS-rollout/DFS or setting larger beam size in BFS. Specifically, the second row of Figure 3 can clearly demonstrate the performance of each tree-search algorithm when given the same token consumption. Basically, MCTS-alpha and MCTS-rollout are still dominant in most cases.
3. At last, we never claim in our paper that MCTS-alpha/rollout is the new SOTA over all evaluation tasks. Our experiments are not designed to show new SOTAs or the most efficient tree search algorithm. **We treat different search algorithms as TS-LLM’s variants and equally present their pros and cons. We believe every algorithm has its limitations and no one can claim superiority in all scenarios.** For example, as we illustrate in our analysis, MCTS, BFS, and DFS generally perform well in shallow search problems while MCTS-α and MCTS-Rollout are dominant in deep search problems.

We thank you for your comment and understand that readers may have similar misunderstandings. To make this more clear, we’ve also modified this and incorporated all three points in our rebuttal revision. Specifically, we modify the Benchmark algorithms subsection in Section 4.1 to include more clarifications among benchmark algorithms and our experiment objectives. We’ve also modified the results analysis in Q1, to incorporate the Path@N result shown in Figure 3 for a better presentation.

Reviewer comment (Evaluation):
In Table 4, why do DFS/BFS/MCTS yield identical results for the RLHF task? This paper should clarify this.

Author response:
Thanks for your comments and suggestions. We do clarify this in our paper in the last two sentences of the first paragraph in Q1’s answer.

Let us clarify this more. When evaluating path@1 performance, we only generate one path given a search algorithm. It means MCTS cannot take the backward step (because it only conducts backward operation after it reaches the terminal node), BFS beam size is 1, and DFS cannot prune node and only get 1 path. In this case, MCTS, BFS, and DFS degenerate to greedy search over value. Results are different on the first 3 tasks because they are using sentence-level node expansion, so the action nodes are based on sampling. For the RLHF task, we derive the node by conducting a forward process and extract the top 50 tokens, it is deterministic as long as the input is the same.

We fully agree that we need more explanations and have modified the analysis results in the first paragraph of our Q1’s answer. We’ve also clarified the Path@1 setting and additionally added a new result analysis by referring to the Path@N result shown in Figure 3. In this setting, MCTS, BFS and DFS will be different so we can conduct a better analysis. We’ve also added more explanations in the caption of Table 2 to better illustrate the reason why DFT/BFS/MCTS yield identical results.

Reviewer comment (Presentation):
In Figure 1, the presentation should be further clarified. The authors did not evaluate Game24 on a token-level, however the figure may mislead readers.

Author response:
Thanks for your comment, we have added the clarification in the footnote of Figure 1.

Reviewer comment (Writing):
Is the term "$k$-node" used in the context of BFS equivalent in meaning to "$k$-node" in the context of MCTS?

Author response:
Exactly. The k is mentioned in the last paragraph of Section 3.1, as the number of nodes we subsample during the leaf node expansion. In other words, it is how many child nodes we have given a parent node.

Reviewer comment (Reproducibility):
Do BFS and DFS utilize the same Large Language Model (LLM) as MCTS-$\alpha$ and MCTS-Rollout, or do they employ GPT-4 as their original Tree of Thought (ToT) configuration?

Author response:
All search algorithms utilize the same LLM, which is LLaMA2-7B in the first three tasks and GPT-2 in the RLHF alignment task. We want to show that our framework is applicable to all different LLMs of any size and directly applying ToT pipeline to them may not work.

Reviewer comment (Experiments):
Based on the result in Table 2, it is unclear if the proposed method is better than the existing method such as BFS (ToT). For instance, in GSM8K, the BFS is better than the proposed methods such as MCTS-\alpha and MCTS-Rollout. We spent a good amount of time getting value functions and the reward model working, but the performance of the proposed method is still worse than the simple baseline. As a result, I am not convinced that this approach will work better or not.

Author response:
Thank you for your comment.  There are three things we want to clarify:
1. **We want to clarify that TS-LLM’s DFS/BFS variants are different from that in Tree of Thought (ToT). They are TS-LLM’s variants instead of ToT baselines.** The main difference is that we use a learned value function instead of prompting advanced LLMs, which is one of our important contributions. We have shown in Table 4 the result of BFS and MCTS-alpha on Game24 by prompting LLaMA2-7B model as value function and ORM. It can exactly be regarded as an instance of ToT over small LLMs. Our conclusion is that prompting a small LLM can not offer accurate value estimation and the performance of tree-search is even worse than simple CoT baseline.
2. Table 2 represents path@1 results. A much more comprehensive result can be seen in our Q4’s experiment. We additionally conduct experiments on search aggregation shown in Figure 3, where you can get Path@N results and scale up your token consumption by running multiple times of search in MCTS/MCTS-alpha/MCTS-rollout/DFS or set larger beam size in BFS. Specifically, the bottom line of Figure 3 can clearly demonstrate the performance of each tree-search algorithm when given the same token consumption. Basically, MCTS-alpha and MCTS-rollout are still dominant in most cases.
3. At last, we never claim in our paper that MCTS-alpha/rollout is the new SOTA over all evaluation tasks. Our experiments are not designed to show new SOTAs or the most efficient tree search algorithm. **We treat different search algorithms as TS-LLM’s variants and equally present their pros and cons. We believe every algorithm has its limitations and no one can claim superiority in all scenarios.**

To make this more clear, we’ve also modified this and incorporated all three points in our rebuttal revision. Specifically, we modify the Benchmark algorithms subsection in Section 4.1 to include more clarifications among benchmark algorithms and our experiment objectives. We’ve also modified the results analysis in Q1, to incorporate the Path@N result shown in Figure 3 for a better presentation.

Reviewer comment (Experiments):
It seems that most of the experiments are about improving the value function and reward function, not about fine-tuning the LLM. For example, if we do fine-tuning Table 2, are we able to improve the performance? Essentially, I want to see more results based on question 5 written in the paper: Can TS-LLM further train LLM? I am happy to increase the score if the answers are answered.

Author response:
We do present experimental results by finetuning Table 2 and Figure 3. The specific results are presented in Table 5 and Table 6. In Table 5, we present how we can leverage tree-search examples in further finetuning the LLM-based value function. In Table 6, we show more comprehensive results over GSM8K and RLHF alignment and present how tree-search examples can increase LLM generation by finetuning LLM policy $\pi_{\theta_0}$ to $\pi_{\theta_1}$, LLM value $V_{\theta_0}$ to $V_{\theta_1}$ and both. Does this address your concerns?

Reviewer comment (Reproducibility):
No details on the value/reward model besides the mention that it is a decoder-only model.

Author response:
**Question 2: No details on the value/reward model besides the mention that it is a decoder-only model.**

As we mentioned in section 3.2.1, the value/reward model architecture is simple, we append a value head (which is an MLP that outputs a scalar, e.g. a linear layer) to a decoder-only transformer. For GSM8k, Game24, and ProntoQA, the decoder-only transformer is the pretrained LLaMA2-7b without the `lm_head` module. For the alignment task, the decoder-only transformer is the pretrained GPT2-small without the same output `lm_head`. And we used the pretrained weights as the initialization.

Reviewer comment (Writing):
Explanation of the algorithms doesn’t clarify when the value/reward models are used.

Author response:
**About the Value function:**

**Question 8: Clarification that the proposed models are implemented as an additional head on top of the same representation. When processing a sample, are the models added to the same architecture, so the new model produces both the new tokens and then estimations? Or are those independent calls with shared weights?**

Thank you for your suggestions. We choose to use the term “LLM-based” value in our paper to help clarify that we use an LLM based value function. We also have made more clarifications in Section 3.2.1 and the model and training detail subsection in Section 4.1. Specifically, we describe that “typically, LLM value’s decoder is adapted from original LLM policy $\pi_\theta$’s decoder, or alternatively, the LLM value $v_\theta$ and policy $\pi_\theta$ can have a shared decoder”. 

Let me clarify our current implementation. In our current implementation, we use a separate LLM policy and LLM value function and their decoders are not shared. This is because our critic learning needs to have training samples from the SFT policy (So our critic can be seen as an on-policy value function for the SFT sampling policy). Our experimental procedure is: We first finetune the base LLM using CoT examples in the training set, to form SFT policy. Then we sample from the SFT policy to construct the critic learning training examples. And finally we get our LLM-based value function by finetuning the base LLM (with additional value heads) using these critic training examples. Though the decoders are not shared among policy and value, they are adapted from the base model. So they may still be quite similar.

And we agree with you that the shared structure might be better for computational efficiency since you do not need to call the model twice and maybe a shared decoder can get rid of all burdens brought by value function evaluation (you can reuse most computation from the policy generation). We also mention it in Appendix D.11 when discussing engineering potentials. There are several ways to achieve it. We can:
1. Instead of using SFT-tuned model, we can use few-shot prompted based model to generate the samples for value function training. And then we train the SFT tuned policy and value function with shared decoders.
2. We have some offline data that help me train the value function in advance.
3. We are doing the training in an online manner so the critic function can be iteratively trained, similar to the setting in AlphaZero.

Currently, we are working on adding a new experiment about the shared decoder by using the same data we train the separate value and policy LLM. We want to show how such model sharing might influence the performance and also efficiency. 

**Question 9: There is no need to test in other LLMs. My point was that the paper should be written as it happened: it was always the same representation, and that’s ok. Cite state of shared weights in AlphaZero and other work.**

Thank you for your suggestions. Please refer to our response in Q8 and we have cited the state of shared weights in AlphaZero (Figure 1.b in AlphaZero’s paper).

Reviewer comment (Presentation):
Some details are buried in the appendix but need to be clearer.

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
1. We’ve corrected the wrong version of the citation of CoRe and cited the self-consistency paper in Section 4.1. 
2. We’ve modified the description of MCTS variants in the paper in Section 3.2.2 and added a comparison of the tree-search algorithms in TS-LLM in Appendix C.2. 
3. We’ve changed the “juxtapose” on page 6 Section 3.4 into a more precise expression. 
4. We’ve defined the state $s$ in Section 3.2.1 outside the transition function $g$. 
5. We’ve defined SFT as “supervised finetuning” in Section 4.1. 
6. We’ve modified the analysis to make it more clear and precise in Q3’s analysis in Section 4.2.
We’ve added the necessary clarification about whether intra- or inter-tree search is used in Table 2 and Table 3 and clarify the detailed setups in Appendix D.5.

Reviewer comment (Reproducibility):
Some settings need to be fully explained and it is unclear what version was used.

Author response:
**About tree-max-width:**

Thank you for making your concern clear. We finally get your point: you are worried about whether the algorithm comparisons are heavily influenced by how we set tree max-width. And you think it is possible that we draw the comparison by carefully tuning these hyperparameters.

Our answer is no. We do not purposely tune this hyperparameter to make an MCTS-like algorithm the best. As we said, in fact we do not care whether MCTS-like algorithms are SOTA or not. We care about equally presenting their pros and cons. The main reason we tune it is not for algorithm comparisons but for the tradeoff between computation and performance for all tree-search algorithms.

And here is our answer to your questions:

**Question 1: Clarify that the search space depends on the parameters max-width and max-depth, and that is not a parameter of the algorithm.**

We make further clarifications and discuss these two algorithm-agnostic parameters and how different action spaces can influence them, in the last paragraph of Section 3.1. We also address the point you mention about Table 1. We move the depth and width hyperparameters to the right and mention that they are hyperparameters for determining search space. 

**Question 2: Clarifying that comparison among search algorithms is better within the same state space, and that different width might lead to different trade-off. No SOTA.**

Thank you for your suggestions. We have updated it in the analysis of Q2.

**Question 3: And, in all the algorithms, does expanding a node/state produce tree-max-width states?**

Not necessarily. As we mentioned in Section 3.1 sentence-level node subsection, for each non-terminal node, the search tree is expanded by sampling tree-max-width actions and dropping the duplicated ones. So the main reason is that we drop the duplicated one.

**Question 4: A table similar to Table 3 but for GSM8k and PrOntoQA.  A more detailed description of the protocol for setting the max-width, including how additional values for max-width are selected for the new variations of table 3.**

Thanks for your comments. We updated the results for GSM8K/PrOntoQA in Appendix D.6. In our submission, for the ablation of tree-max-width on Game24, we did not test it on 3 seeds, but now we’ve added the results of all 3 tasks on 3 seeds and report the average value and standard deviation in Table 7,8,9 and we modified the value of Table 3 to the mean value on Table 8.

We also add more analysis about the results and as well as the protocol about how we select the width parameters. Please refer to Appendix D.6 for them.

**Question 5: Table 3 has no variance but the appendix says that 3 seeds were used.**

Thank you for pointing this out. This is because we conduct 3 seeds experiment for Table 2 but for Table 3 it is mainly 1 seed result. To address this, we also update the results in Table 3 (only report mean because of space limitation) and refer to Appendix D.6 for full 3 seeds results (mean an std in Table 7,8,9).

Reviewer comment (Reproducibility):
The appendix says that sometimes intra or inter-tree search is used, but it’s not clear which experiment uses which setting.

Author response:
We’ve added the necessary clarification about whether intra- or inter-tree search is used in Table 2 and Table 3 and clarify the detailed setups in Appendix D.5.

Reviewer comment (Reproducibility):
No discussion on how the hyper-parameters were set.

Author response:
**Other topics:**

**Question 10:  Make sure that for all the hyper-parameters, the protocol for selection is discussed, as well as the sensibility to the parameter.**

We add a new subsection in Appendix D.10, to fully discuss our protocol for choosing all tree-search-related hyperparameters in our paper.

Reviewer comment (Reproducibility):
The lack of hyper-parameter discussion is especially critical for MCTS, as it might be very sensitive to them.

Author response:
**Question 11: How were the hyper-parameters setup. How much budget was used to set these numbers? How sensitive are the results to the particular parameters?**

During our experiments, we found that the performance and token consumption are sensitive to several hyperparameters: first is the general hyperparameters of building the search trees, e.g. tree-max-depth and tree-max-width, for the former, we choose the max-depth according to the statistics of the distribution of the number of steps from the dataset sampled by the LLM policy on the training set; while for the latter we’ve shown in Table 3 that there is a trade-off between token consumption and the size of the search space.
MCTS variants (MCTS-alpha, MCTS-rollout and MCTS) are also sensitive to hyperparameters, we keep most hyperparameters as the default values used in AlphaZero and MuZero except for $c_{init}$ which we select a value from two possible ones {0.3, 3.0} when computing $c_{puct}$ to control exploration and exploitation in MCTS variants.
For the Dirichlet noise, this might be important since there is some guess that DeepMind sets this hyperparameter according to the statistics of the dataset collected for each task. However, due to the limit of computation resources, we just adopted the default value in AlphaGo as 0.3, which is specified for chess (refer to Page 14 in the AlphaZero paper (https://arxiv.org/pdf/1712.01815.pdf) and Appendix B in MuZero(https://www.nature.com/articles/s41586-020-03051-4)).

Reviewer comment (Reproducibility):
Many other details, such as the 0.3 in the Diriletch distribution for adding noise, are important.

Author response:
**Question 11: How were the hyper-parameters setup. How much budget was used to set these numbers? How sensitive are the results to the particular parameters?**

During our experiments, we found that the performance and token consumption are sensitive to several hyperparameters: first is the general hyperparameters of building the search trees, e.g. tree-max-depth and tree-max-width, for the former, we choose the max-depth according to the statistics of the distribution of the number of steps from the dataset sampled by the LLM policy on the training set; while for the latter we’ve shown in Table 3 that there is a trade-off between token consumption and the size of the search space.
MCTS variants (MCTS-alpha, MCTS-rollout and MCTS) are also sensitive to hyperparameters, we keep most hyperparameters as the default values used in AlphaZero and MuZero except for $c_{init}$ which we select a value from two possible ones {0.3, 3.0} when computing $c_{puct}$ to control exploration and exploitation in MCTS variants.
For the Dirichlet noise, this might be important since there is some guess that DeepMind sets this hyperparameter according to the statistics of the dataset collected for each task. However, due to the limit of computation resources, we just adopted the default value in AlphaGo as 0.3, which is specified for chess (refer to Page 14 in the AlphaZero paper (https://arxiv.org/pdf/1712.01815.pdf) and Appendix B in MuZero(https://www.nature.com/articles/s41586-020-03051-4)).

Reviewer comment (Reproducibility):
Knowing the number of roll-outs would be useful.

Author response:
**Question 3: No specific statistics about the behaviour of the algorithms beyond the aggregated number of tokens. For instance, knowing the number of roll-outs would be useful.**

Thank you for your comment. We want to confirm the meaning of ‘roll-outs’ here. If it means the number of sequences a certain search algorithm outputs, we’ve already shown it in the first row of Figure 3 and Appendix D.6. We hope for your further explanation about the definition of ‘roll-outs’ and we will try to show you the results.

Reviewer comment (Evaluation):
While the paper discussed implementations, it’s not clear what the actual performance might be; even though the implementation can be challenging and improved if this method becomes more popular, we need to know about the current cost of solving the tasks.

Author response:
**Question 4: No discussion of the wall-time for decoding**

Thank you for your comment. We’ve added different algorithms’ decoding wall-time of GSM8k, Game24 and ProntoQA in Table 13~15 in Appendix D.9. The results demonstrated the gap of computational efficiency between CoT/CoT-SC decoding and tree-search algorithms in TS-LLM, due to the complicated search procedures and extra computation introduced by calling value functions in the intermediate states. 

We totally agree that computational efficiency is an important issue and address it in Appendix B. But note that our current implementation is just an algorithm prototype without any engineering acceleration. We are continuously working on accelerating our tree search framework codebase including caching past key/values on the tree to eliminate repeated computations and offloading tensors on RAM, etc.

Reviewer comment (Experiments):
Keeping the tree in GPU memory can be expensive, and moving tensors in and out could reduce GPU utilization.

Author response:
**Other topics:**

**Question 11: Discussion about engineering challenge.**

Thank you for your suggestion. We added 5 points in Appendix D.11 (previous D.9) about the engineering challenges and how we can further increase the efficiency with more engineering efforts.

Reviewer comment (Evaluation):
Analysis can be improved.

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
...
6. We’ve modified the analysis to make it more clear and precise in Q3’s analysis in Section 4.2.

Reviewer comment (Reproducibility):
Given the lack of details on how the hyper-parameters were set, and how sensitive the results are to them, it’s hard to qualify the contribution, and how likely these algorithms are to perform well in other domains. This perception can change depending on the answer of the authors.

Author response:
**Question 11: How were the hyper-parameters setup. How much budget was used to set these numbers? How sensitive are the results to the particular parameters?**

During our experiments, we found that the performance and token consumption are sensitive to several hyperparameters: first is the general hyperparameters of building the search trees, e.g. tree-max-depth and tree-max-width, for the former, we choose the max-depth according to the statistics of the distribution of the number of steps from the dataset sampled by the LLM policy on the training set; while for the latter we’ve shown in Table 3 that there is a trade-off between token consumption and the size of the search space.
MCTS variants (MCTS-alpha, MCTS-rollout and MCTS) are also sensitive to hyperparameters, we keep most hyperparameters as the default values used in AlphaZero and MuZero except for $c_{init}$ which we select a value from two possible ones {0.3, 3.0} when computing $c_{puct}$ to control exploration and exploitation in MCTS variants.
For the Dirichlet noise, this might be important since there is some guess that DeepMind sets this hyperparameter according to the statistics of the dataset collected for each task. However, due to the limit of computation resources, we just adopted the default value in AlphaGo as 0.3, which is specified for chess (refer to Page 14 in the AlphaZero paper (https://arxiv.org/pdf/1712.01815.pdf) and Appendix B in MuZero(https://www.nature.com/articles/s41586-020-03051-4)).

Reviewer comment (Writing):
The notion of aggregation is misleading in Section 3.2.3, as aggregation refers to using a set of things to get something new, but in this case, we are just selecting an output, not combining them.

Author response:
**Question 1: The notion of aggregation is misleading (Sect 3.2.3). Aggregation refers to using a set of things to get something new, but in this case, we are just selecting an output, not combining them.**

We totally understand your concern that the notion of ‘aggregation’ might be misleading. While we adopted it from the previous papers CoT-SC(https://arxiv.org/abs/2203.11171) and   RAP(https://arxiv.org/abs/2305.14992) for consistent terminology.

Reviewer comment (Presentation):
The most relevant work is in the appendix — on page 13 the most relevant work, CoRe (Zhu et al., 2022), proposes to finetune both the reasoning step generator and learned verifier for solving math word problems using MCTS for reasoning decoding — please fix the reference to ACL 2023.

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
1. We’ve corrected the wrong version of the citation of CoRe and cited the self-consistency paper in Section 4.1.

Reviewer comment (Experiments):
Given that MCTS combines both the policy and the value, perhaps a comparison with BFS or DFS is not the more meaningful; algorithms based on A* rely on the accumulated cost g — in this case inverse to the likelihood of the prefix — and h, an heuristic estimating the value of the rest of the path. While the value function in the paper does not take into account the length, it’s still possible to consider their combination.

Author response:
**Question 5: Given that MCTS combines both the policy and the value, perhaps a comparison with BFS or DFS is not the more meaningful. Algorithms based on A\* rely on the accumulated cost g —in this case inverse to the likelihood of the prefix— and h, an heuristic estimating the value of the rest of path. While the value function in the paper does not take into account the length, it’s still possible to consider their combination.**

Yes, we agree with your ideas. This can actually be viewed as a dense reward setting in which, at each step, there is a nontrivial reward. It is natural and heartening to extend our method to this setting like extending from AlphaZero(https://arxiv.org/pdf/1712.01815.pdf) to MuZero(https://www.nature.com/articles/s41586-020-03051-4).

Reviewer comment (Writing):
Please define the state \( s \) by itself, not inside \( g \) — on page 4: given state s and action a, is known as: g( s = (x 0:L−1, y 0:t−1), a = y t) = (x 0:L−1, y 0:t).

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
4. We’ve defined the state $s$ in Section 3.2.1 outside the transition function $g$.

Reviewer comment (Writing):
Why is \( g \) defined on page 4? Section 3.1 uses \(\phi_\theta\).

Author response:
**Question 7: Why defining $g$ in page 4? Section 3.1 uses \phi_theta**

We guess you mean $\pi_\theta$ in Section 3.1. They have different meanings here, $\pi_\theta$ is the policy that takes an input $x$ and outputs actions. While $g$ is defined in the dynamics transition function of the problem, for the tasks we are testing now the transition function is deterministic, i.e. appending output tokens to the prefix/prompt string. The transition function $g$ can be considered being influenced by the LLM policy $pi_\theta$ as $g(s, a) = g(s, a \sim \pi_\theta(\cdot | s))$

Reviewer comment (Writing):
On page 6, “This helps us juxtapose the performance” — is juxtapose the right word? Perhaps it’s a meaningful comparison.

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
3. We’ve changed the “juxtapose” on page 6 Section 3.4 into a more precise expression.

Reviewer comment (Writing):
On page 7, “For value and ORM training, the data are generated by randomly sampling the SFT policy’s rollouts on the training set” — please define SFT, which is likely supervised fine-tuning.

Author response:
According to your suggestion, we’ve made the following modifications and clarifications to our paper to address the writing issue you mentioned:
5. We’ve defined SFT as “supervised finetuning” in Section 4.1.

Reviewer comment (Experiments):
This paper needs to have more experiments to show the improvement. For example, more models and more tasks. For more tasks, maybe you can find some here “https://github.com/Ber666/llm-reasoners”.

Author response:
Thank you for your suggestions. In fact, we do refer to “https://github.com/Ber666/llm-reasoners” for some of our experiments including GMS8K, ProntoQA and Game24. We also add a brand new RLHF task to demonstrate the general applicability of our TS-LLM framework. You can have a more detailed look at their repo and it turns out that most environments they provide do not contain any benchmark results (the table in their Experiment Results section in README).

Reviewer comment (Evaluation):
The paper should discuss which kinds of tasks are better for using MCTS rollout and which are better for MCTS alpha.

Author response:
Thank you for your comment. The only difference between MCTs-Rollout and MCT-alpha is that MCTS-rollout conducts the search from the beginning while MCTS-alpha conducts the search from the last search node. We think MCTS-rollout is just an offline version of MCTS-alpha since it reconducts the full search from the beginning every time. Our conclusion is, basically MCTS-rollout and MCTS-alpha will be good at the same kind of task, while MCTS-rollout can scale up token consumption to search for better generation. According to your suggestion, we have added a discussion to the MCTS-Rollout description in section 3.2.2.

Reviewer comment (Experiments):
Have you tried using both rollout and value together? If the speed of rollout and the speed of the value network are very different, you can reference the paper “Multiple Policy Value Monte Carlo Tree Search.”

Author response:
Thank you for your comment and we will refer to this paper. Currently, we use two separate networks to conduct rollout and value generation respectively and the speed of rollout and the speed of value estimation are similar. But note that we can also utilize a shared LLM decoder for both policy and value network so the rollout and value estimation can be conducted simultaneously and be more efficient.