Paper: Making PPO even better: Value-Guided Monte-Carlo Tree Search decoding (ICLR-2024)

Reviewer comment (Evaluation):
One of the major weaknesses is the inference time of PPO-MCTS. As the authors said, "... PPO-MCTS is 2S times slower due to the tree construction, where S is the number of simulations ...". It is questionable that this method can be used in practice.

Author response:
**Inference-time efficiency:** The significant inference-time overhead is indeed a weakness of our method, which we have acknowledged in the “Limitations” section. We argue that PPO-MCTS may be found useful in several scenarios:
    * When the quality of generated text is critical, PPO-MCTS offers a way to improve text quality out of existing trained models, at some expense of additional inference-time compute;
    * Our method shows the promise of using MCTS to explore and rollout better trajectories during PPO training (improving upon the commonly-used temperature-sampling rollout), which we briefly mentioned in the “Conclusions and Future Work” section. This has the potential to improve PPO training itself, and will not incur additional inference-time latency (assuming not using MCTS for inference). To achieve this, our proposed PPO-MCTS decoding method would be an indispensable component.

Reviewer comment (Experiments):
In Table 1, "PPO (4x more steps)" outperforms "PPO" (75.50 vs. 52.44), and the result is getting close to PPO-MCTS. Does "PPO (4x more steps)" converge? Why not try more steps for PPO, e.g. PPO (8x more steps)? In addition, how is the performance of PPO(4x more steps)-MCTS?

Author response:
**PPO with even more steps:** Thanks for your suggestion, we added more experimental results to Table 6 (rows “PPO(9x more steps)”, “PPO(4x more steps)+MCTS”, and “PPO(9x more steps)+MCTS”). PPO converged at “9x more steps”, and yet its goal satisfaction rate (86.34) is still lower than PPO+MCTS (86.72). MCTS is able to improve the goal satisfaction rate by at least 10 points across all these PPO models.

Reviewer comment (Experiments):
Since the PPO-MCTS is much slower than PPO, it would be better to evaluate PPO-MCTS with 1-50 simulations instead of only 10, 20, and 50 simulations. This result can provide readers with a tradeoff between performance and evaluating time.

Author response:
**MCTS with fewer simulations:** We added more experimental results to Table 5, with the number of simulations $S=\{1,2,5\}$ (in addition to existing results where $S=\{10,20,50\}$). (Note that $S=1$ reduces MCTS to greedy decoding, and thus has very low diversity metrics.) We do observe that the goal satisfaction rate steadily increases as $S$ increases.

Reviewer comment (Writing):
The description of the MCTS variant is somewhat difficult to understand.

Author response:
**Clarity:** Hopefully our response point #3 above addresses your comment that our description of MCTS “is somewhat difficult to understand”, since aside from the modifications we carefully described, the MCTS we use is standard as in prior literature. If there’s anything else we can make it easier to follow, please don’t hesitate to let us know ;)

Reviewer comment (Novelty):
In the presentation of the new MCTS, there are some alterations highlighted, but the novelty of most of these changes is uncertain.

Author response:
**Novelty:**
    * **Replacing V with Q:** Thanks for pointing out the original source of this idea. Other reviewers (GmAu and ZQMA) also agree with this. We added a citation to Kocsis and Szepesvari (2006), and removed this point from our novelty claim.
    * **The KL term:** We appreciate your perspective, but we respectfully hold a different viewpoint on this. The goal of search is *not* to make the KL term vanish – the KL term is conventionally included in the PPO training objective, and in MCTS it is a part of the value/Q functions that guides the search. Also, the KL term steers decoding towards the reference policy, while the prior in Eq 3 makes the decision informed by the PPO policy (which is regularized by the KL term to stay close to the reference policy during PPO training).
    * **Initializing Q with V:** Based on our investigation of several MCTS implementations as listed below,  this technique is not adopted, so we concluded that it is *not* standard for MCTS. If you can share evidence that shows otherwise, we’d greatly appreciate it.
[[1](https://github.com/google-deepmind/open_spiel/blob/6e46a2d5746250cddb5bcc3124c73415f0c18732/open_spiel/algorithms/mcts.cc#L366-L428), [2](https://github.com/NohTow/PPL-MCTS/blob/ed588ca8f8abf4dd113f57d6c95c804ef9ca961e/mcts_rollout.py#L486-L521), [3](https://github.com/ImparaAI/monte-carlo-tree-search/blob/469ef42a1dc49aa1dbe4e6fd16bef05c200267d7/montecarlo/montecarlo.py)]
    * **Forbidding exploration after terminal states:** Thanks for pointing out that this is standard practice. Reviewer (GmAu) also agrees with this. We removed this point from our novelty claim.

Reviewer comment (Theory):
Regarding "replacing V with Q": even the original UCT paper defined the problem as an MDP with non-terminal rewards included, and using Q(s,a) is much more standard than V(s), which has an after-state flavor. Using the KL penalty term as (or in the) reward seems a bit odd, as it steers the search toward the original policy, which may not be a bad thing, but this is already done by the use of the prior in equation (3). The KL term not vanishing with additional search means that the MCTS algorithm does not converge asymptotically to an optimal action.

Author response:
**Novelty:**
    * **Replacing V with Q:** Thanks for pointing out the original source of this idea. Other reviewers (GmAu and ZQMA) also agree with this. We added a citation to Kocsis and Szepesvari (2006), and removed this point from our novelty claim.
    * **The KL term:** We appreciate your perspective, but we respectfully hold a different viewpoint on this. The goal of search is *not* to make the KL term vanish – the KL term is conventionally included in the PPO training objective, and in MCTS it is a part of the value/Q functions that guides the search. Also, the KL term steers decoding towards the reference policy, while the prior in Eq 3 makes the decision informed by the PPO policy (which is regularized by the KL term to stay close to the reference policy during PPO training).

Reviewer comment (Novelty):
Regarding "initializing Q with V": this is probably the standard way of using value functions in MCTS.

Author response:
**Novelty:**
    * **Initializing Q with V:** Based on our investigation of several MCTS implementations as listed below,  this technique is not adopted, so we concluded that it is *not* standard for MCTS. If you can share evidence that shows otherwise, we’d greatly appreciate it.
[[1](https://github.com/google-deepmind/open_spiel/blob/6e46a2d5746250cddb5bcc3124c73415f0c18732/open_spiel/algorithms/mcts.cc#L366-L428), [2](https://github.com/NohTow/PPL-MCTS/blob/ed588ca8f8abf4dd113f57d6c95c804ef9ca961e/mcts_rollout.py#L486-L521), [3](https://github.com/ImparaAI/monte-carlo-tree-search/blob/469ef42a1dc49aa1dbe4e6fd16bef05c200267d7/montecarlo/montecarlo.py)]

Reviewer comment (Novelty):
Regarding "Forbidding node expansion after terminal states": this is just the normal thing to do, and probably there is no MCTS implementation that would expand the tree beyond a terminal state.

Author response:
**Novelty:**
    * **Forbidding exploration after terminal states:** Thanks for pointing out that this is standard practice. Reviewer (GmAu) also agrees with this. We removed this point from our novelty claim.

Reviewer comment (Reproducibility):
The backup rule, as defined, looks somewhat strange. The update (5) leads to NaN if any of the child nodes (s') is not yet sampled (N(s')=0). The usual backup rule is to update it with the return from the current trace. Maybe it is happening here as well, but it is awkwardly written. It would be useful to describe the algorithm with pseudocode so that similar issues can be clarified better.

Author response:
**Backup algorithm:** We apologize for the confusion. We improved the notation in Equations 4-6, and added an algorithm (Algorithm 1, in Appendix A.3) to clarify the backup process of MCTS. To answer your question, a NaN result cannot happen, because to get a zero denominator you need $N(s’) = 0$ for all $s’$ (where $s’$ is a child node of state $s$, where the child edge (a.k.a., the action taken) is $a$). At least one of the $N(s’)$ must be non-zero, because $s$ is an intermediate node and the “expand” stage of the current simulation round has explored one of its subtrees.

Reviewer comment (Presentation):
My first criticism regards the name of the submission and method. I'm not totally in agreement that the submission is about "Making PPO Better". The submission is about using MCTS to better align the model's output with human feedback. That the value function comes from PPO in particular seems not very material to the actual contribution of the submission. I also find the name PPO-MCTS somewhat misleading. To me, this evokes a method that combines PPO and MCTS in the spirit of PG Search (https://arxiv.org/abs/1904.03646) or RL Search (https://openreview.net/forum?id=D0xGh031I9m), which is not what the submission intends to convey. It would be more clear to label the approach PPO+MCTS.

Author response:
**Title and method name:** We agree that our proposed method is an inference-time algorithm that complements PPO, rather than improving PPO itself. We will change our title to **Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding**. We will also change the method name to **PPO+MCTS** to better reflect that we are combining two existing approaches.

Reviewer comment (Novelty):
My second criticism regards the submission's claims of novelty regarding MCTS point 1, "Replacing the edge values V with the Q-function to ensure consistency with PPO training." The description in the introduction is somewhat unclear but clarified under equation (3). However, this is not novel — it was actually the original formulation of MCTS. See *Bandit based Monte-Carlo Planning* (Kocsis and Szepesvari, 2006), which the submission neglects to cite.

Author response:
**Replacing V with Q:** Thanks for pointing out the original source of this idea. Other reviewers (Svcj and ZQMA) also agree with this. We added a citation to Kocsis and Szepesvari (2006), and removed this point from our novelty claim.

Reviewer comment (Theory):
Regarding the submission's claimed novelty point 2, "Initializing the Q of children actions from the V of their parent node to encourage exploration," the given explanation does not make sense. The submission argues that large scales of Q in PPO models suppress exploration, but the scale of Q should not matter. Any increase in the scale of Q can be mitigated by increasing the scale of c_puct.

Author response:
**The scale of Q and c_puct:** The unknown scaling factor in Q cannot be mitigated by tuning $c_\text{puct}$, because the unknown scaling factor is instance-specific (Appendix A.5). Empirically, we added experiments with different values of $c_\text{puct}$ while turning off the “initialize Q with V” technique: while increasing $c_\text{puct}$ does mitigate the exploration degeneracy, it causes degradation of task performance. For details please see additions to Table 5 (Appendix D).  

Apologies for the confusion. As we discussed in Appendix A.5, Reward whitening "[...] introduces a batch-specific scaling factor that is unknown to us at inference time. Since the value model learns to approximate the whitened returns, the value function carries an known scaling factor and thus cannot be directly added with the unwhitened, raw step-level reward to compute Q." The batch-specific scaling factor in value function V cascades to an instance-specific scaling factor of Q, and thus $Q(s, a)$ (or $V(s)$, if we do the "approximate Q with V" technique) carries an implicit, unknown scale that is specific to the instance $(s, a)$.

Reviewer comment (Presentation):
My third criticism is structural. The submission's method section mostly just explains MCTS, which is not new methodology from the paper — it is background material. The submission should describe MCTS as background material in the preliminaries section. The methodology section should only explain how the submission applies MCTS in its specific context. It is acceptable if the methodology section is short.

Author response:
**Structural:** Thanks for the suggestion. We believe it is better to keep the introduction of MCTS algorithm in the Method section for sake of fluency in narration, because there are several parts tailored to PPO models and language generation tasks.

Reviewer comment (Novelty):
The citation used to justify not doing Monte Carlo rollouts for efficiency considerations ("Following Chaffin et al. (2021), we do not do Monte-Carlo rollout due to efficiency considerations") is not appropriate. Using a value function instead of Monte Carlo rollouts is common practice going back at least to AlphaGo Zero.

Author response:
**Using value function in place of Monte-Carlo rollout:** Indeed we can confirm that AlphaGo-Zero already uses this technique, and we added a citation there (in addition to Chaffin et al. (2021)).

Reviewer comment (Experiments):
The main weakness of the paper is the missing comparison axis of efficiency in the paper and the baselines it therefore decides to evaluate against. The fundamental issue is that RLHF is only used because it is a more efficient alternative to best-of-n decoding with a reward model. As shown in [Gao et al (2022)](http://arxiv.org/abs/2210.10760) as well as [Stiennon et al (2020)](http://arxiv.org/abs/2009.01325), just decoding \(n\) samples from a model and choosing the best one using a reward model ("best-of-n") is as performant as RLHF and closer in KL to the original model. The reason this method isn't used is because it can require incredibly high \(n\) and be computationally infeasible at scale. This paper proposes to do a computationally expensive method after training without really tackling the issue of computational cost. At minimum, the authors need to show how their method performs across different computation scales (values of \(S\)) and compare it to PPO, PPO+best-of-n, as well as best-of-n without PPO. The goal should be to give practitioners an idea of the computation/performance tradeoff of this method. The only results on a large-scale task, HH-RLHF, don't seem to be very strong, which casts some doubt on the efficacy.

Author response:
**Efficiency as evaluation axis:** To investigate the efficiency-performance tradeoff, we added more experimental results to Table 5, and now it reports performance with the number of simulations $S=\{1,2,5,10,20,50\}$, representing increasing amounts of compute. We do observe that the goal satisfaction rate steadily increases with more inference-time compute.

Reviewer comment (Experiments):
Given the relative simplicity of the idea, there need to be more baselines to convincingly show that (1) the value model is necessary and (2) that MCTS is the best way to use it.

Author response:
**More experiments on best-of-n (with reward model):** Thanks for your suggestion, we added more experimental results to Table 6 (rows “PPO + best-of-$n$[R]” and “no PPO + best-of-$n$[R]”). PPO + best-of-$n$[R] has much lower performance than our PPO-MCTS (72% vs 87%). From these results, we can see that best-of-$n$ is not as good as MCTS, and thus MCTS is needed.

Reviewer comment (Experiments):
For demonstrating that the value model is necessary, the existing experiment doing MCTS with a reward model is a good start but is not fully convincing. A baseline of best-of-n (with/without PPO) using the reward model should be included.

Author response:
**More experiments on best-of-n (with reward model):** Thanks for your suggestion, we added more experimental results to Table 6 (rows “PPO + best-of-$n$[R]” and “no PPO + best-of-$n$[R]”). PPO + best-of-$n$[R] has much lower performance than our PPO-MCTS (72% vs 87%). From these results, we can see that best-of-$n$ is not as good as MCTS, and thus MCTS is needed.

Reviewer comment (Experiments):
For demonstrating that MCTS is the best way to use the value model, common alternatives should be compared, such as beam search and beam search using the value model.

Author response:
**Comparison with beam search:** Since diversity is an integral part of the *sentiment steering* task, beam search is not applicable there. We reported a baseline method similar to beam search, referred to as “stepwise-value” in Table 6 and also the 3rd paragraph of Sec 5.1. This method evaluates candidate tokens using the value function and keeps more promising tokens (as would beam search do), and supports diversity by sampling from the (softmax of) value function. As reported in Table 6, MCTS is substantially better than this stepwise decoding method.

Reviewer comment (Experiments):
The suggested baselines (best-of-n with/without PPO using the reward model, beam search, and beam search with the value model) are a bare minimum, especially given the large literature on language model decoding. For comparison, [Leblond et al (2021)](https://arxiv.org/pdf/2104.05336.pdf), which the authors note is closely related to their own work, compared against all of the above as well as greedy decoding and MCTS with rollouts.

Author response:
**More experiments on best-of-n (with reward model):** Thanks for your suggestion, we added more experimental results to Table 6 (rows “PPO + best-of-$n$[R]” and “no PPO + best-of-$n$[R]”). PPO + best-of-$n$[R] has much lower performance than our PPO-MCTS (72% vs 87%). From these results, we can see that best-of-$n$ is not as good as MCTS, and thus MCTS is needed. **Comparison with beam search:** Since diversity is an integral part of the *sentiment steering* task, beam search is not applicable there. We reported a baseline method similar to beam search, referred to as “stepwise-value” in Table 6 and also the 3rd paragraph of Sec 5.1. This method evaluates candidate tokens using the value function and keeps more promising tokens (as would beam search do), and supports diversity by sampling from the (softmax of) value function. As reported in Table 6, MCTS is substantially better than this stepwise decoding method.

Reviewer comment (Novelty):
The authors seem to imply that they have a novelty in MCTS by using a Q-value instead of a Value, but this formulation is used by Leblond et al (2021).

Author response:
**Novelty of “Replacing V with Q”:** Thanks for letting us know that this was adopted by Leblond et al. As other reviewers (GmAu and Svcj) pointed out, this technique was also adopted in the original formulation of MCTS. Therefore, we removed this point from our novelty claim.

Reviewer comment (Presentation):
The paper spends too much time on introduction, explaining existing ideas (e.g., MCTS) and notation. For example, Paragraph 4 describing Figure 1 in detail is unnecessary and should be captured by the caption.

Author response:
**Stylistic:** The caption of Figure 1 and the paragraph 4 of the intro are somewhat complementary, and we kept them for completeness of the narration.

Reviewer comment (Experiments):
The main issues preventing recommendation for acceptance are the lack of evaluations that take into account efficiency and the absence of baselines showing that both the value model and MCTS are necessary. Demonstrations are needed showing in which computational regimes the method outperforms best-of-n with/without PPO using the reward model and including those plus beam search baselines on at least one benchmark (e.g., toxicity reduction would be sufficient).

Author response:
**Efficiency as evaluation axis:** To investigate the efficiency-performance tradeoff, we added more experimental results to Table 5, and now it reports performance with the number of simulations $S=\{1,2,5,10,20,50\}$, representing increasing amounts of compute. We do observe that the goal satisfaction rate steadily increases with more inference-time compute. **More experiments on best-of-n (with reward model):** Thanks for your suggestion, we added more experimental results to Table 6 (rows “PPO + best-of-$n$[R]” and “no PPO + best-of-$n$[R]”). PPO + best-of-$n$[R] has much lower performance than our PPO-MCTS (72% vs 87%). From these results, we can see that best-of-$n$ is not as good as MCTS, and thus MCTS is needed. **Comparison with beam search:** Since diversity is an integral part of the *sentiment steering* task, beam search is not applicable there. We reported a baseline method similar to beam search, referred to as “stepwise-value” in Table 6 and also the 3rd paragraph of Sec 5.1. This method evaluates candidate tokens using the value function and keeps more promising tokens (as would beam search do), and supports diversity by sampling from the (softmax of) value function. As reported in Table 6, MCTS is substantially better than this stepwise decoding method.

Reviewer comment (Experiments):
Clarification is needed on why nucleus sampling is used in HH. This seems like an important choice, and previous work has used regular sampling or greedy decoding.

Author response:
**Nucleus sampling in HH-RLHF:** We use nucleus sampling for the HH-RLHF task because, we experimentally found that using greedy decoding results in very verbose and repetitive outputs and has slightly lower reward, and temperature sampling can be unrobost due to the long-tail distribution issue (Holtzman et al. 2020).

Reviewer comment (Reproducibility):
Clarification is needed on what exactly PPO+best-of-n is. Is it decoding \(n\) samples and then choosing the best sample according to a reward model? If so, why is the value model used instead of the reward model?

Author response:
**PPO+best-of-n, value model:** Yes, it means decoding $n$ samples and then choosing the best sample according to the PPO value model. We use the value model instead of reward model because, in some practical use cases, the reward model may be unavailable (e.g., in the toxicity reduction task, maybe PerspectiveAPI is offline or shut down). PPO-MCTS does not rely on the reward model, when using approximations (Appendix A.4).

Reviewer comment (Evaluation):
An explanation is needed for why the Fluency numbers are so different from the original results in Quark.

Author response:
**Fluency numbers:** Our fluency numbers look different than Quark’s because Quark used an *incorrect* perplexity formula (confirmed via their [code](https://github.com/GXimingLu/Quark/blob/30c9a0fe022b7d006b826abec2eb5d3ac288e637/main.py#L322) and by Quark’s author). Their PPL is the inverse probability of the *full generated sequence*, whereby the correct PPL should have a length-normalization factor. Also, single-digit PPLs on text generated by an LM (as in our results) are more plausible than double-digit numbers.