Paper: Learning to Generate Better than your Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The benchmarks used are somewhat lacking. For text generation in 2023, most comparisons for state of the art methods are made by comparisons to other LLMs because fixed metrics have been found to be lacking. A comparison using a framework like Chatbot Arena (Zheng et al. 2023) would have made the results significantly more convincing. These often use GPT-4 as an evaluator, which has now been shown to be provisionally better than traditional metrics (Liu et al 2023, Min et al 2023, Zhou et al 2023, inter alia).

Author response:
__Chatbot Arena with GPT4 Evaluation__

Thank you for your suggestions. We would like to clarify the scope of our paper:  we focuses on RL finetuning of specific downstream tasks (e.g., generating positive reviews or reddit post summary) instead of generating a general purpose chat model. None of our tasks were for dialogue and general chat/instruction following and thus we don’t find Chatbot Arena to be a relevant evaluation platform for this work. Our evaluation and tasks closely follow other published RLHF papers in 2023, e.g., RL4LM [1],  which is published in ICLR 2023. 

Also we aimed to use open-source models for both training and evaluation of our models. Open-source models remain publicly available, guaranteeing the reproducibility of our experiments and results. Whereas closed source models access can be depreciated [2] at any time, making it impossible to reproduce our results. 

[1] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, Ramamurthy et al. 2023

[2] https://openai.com/blog/gpt-4-api-general-availability

Reviewer comment (Evaluation):
The fact that perplexity goes up but output perplexity goes down on IMDB is worrying. I can understand why perplexity would go up: the model is likely collapsing onto a smaller subspace of possibilities, which may still be higher quality. However, the fact that the output perplexity also goes down is worrying: it indicates that after optimization the resultant model is more predictable to LMs which is usually not a good sign for tasks that LMs are bad at in the first place. The results on TL;DR are similarly quite small.

Author response:
__Perplexity Goes up but Output Perplexity does down__

We would like to clarify that perplexity is a measure of fluency rather than a measure of how well a generation does on a task. Our evaluation model for output perplexity is GPT2-large. While we agree GPT2-large is not good at the task in terms of optimizing the reward, it is still a valid model that has been widely used to evaluate whether or not the generation is natural and fluent (e.g., see [1-10]). 

The overall objective that we are optimizing for is $J_{ppo}$, which includes three terms: reward, KL-constraint (i.e., a fluency constraint), and MLE-constraint (i.e., a fluency constraint). We attempted to capture each term’s significance during evaluation through reward, output perplexity, and perplexity respectively. Both output perplexity and perplexity are not perfectly aligned with the terms in the objective but are somewhat correlated. The policy that we are computing a KL-constraint with respect to is different from the output perplexity policy. Furthermore, we would like to point out that there could be high-scoring generations with low fluency as well as high-scoring generations with high fluency. The output perplexity being better after the RL procedure shows that we are maintaining fluent generations while improving the reward that we are optimizing on.

[1] DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts by Liu et al. 2021

[2] PLUG AND PLAY LANGUAGE MODELS: A SIMPLE APPROACH TO CONTROLLED TEXT GENERATION by Dathathri et al. 202

[3] Plug-and-Blend: A Framework for Plug-and-play Controllable Story Generation with Sketches by Lin et al. 2021

[4] POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training by Zhang et al. 2020

[5] COCON: A SELF-SUPERVISED APPROACH FOR CONTROLLED TEXT GENERATION by Chan et al 2021

[6] FUDGE: Controlled Text Generation With Future Discriminators by Yang et al. 2021

[7] Mix and Match: Learning-free Controllable Text Generation using Energy Language Models by Mireshghallah et al. 2022

[8] A DISTRIBUTIONAL APPROACH TO CONTROLLED TEXT GENERATION Khalifa et al. 2021

[9] On Learning Text Style Transfer with Direct Rewards by Liu et al 2021

[10] MASKGAN: BETTER TEXT GENERATION VIA FILLING IN THE ___ Fedus et al. 2018

Reviewer comment (Theory):
On page 6 you write “While we do not expect the SFT policy π g is as good as the optimal π ⋆ , it is reasonable to expect that d π g provides coverage to d π ⋆ .” Why is this reasonable to expect? It is not a prior clear to me that language models cover the space of optimal solutions well, rather than only covering a small percentage of them. Where is the evidence for this claim?

Author response:
**$\pi_{SFT}$  reasonably covers  $\pi^{*}$**

Here we assume $\pi^*$ as the policy that generates the references in the dataset. Then under the assumption that $\pi_{SFT}$ is well trained on the task with references as labels, we claim that $\pi_{SFT}$ has non-zero support over the space of generations that $\pi^*$ may cover. This claim is empirically supported by the fact that SFT has bounded non-infinity validation perplexity: given the definition of perplexity, the only way we can have bounded validation perplexity is for our model to have non-zero probability of generating the references. In other words, this means that the SFT model does have non-zero probabilities generating the references in the dataset.

Reviewer comment (Experiments):
On page 7 you write “For training supervised SFT baselines, we consider only the examples with positive labels.”—This is a somewhat unorthodox decision, as it gives little for the model to contrast with or for the user to control at inference time. It basically reduces this to a finetuning problem. Why was this choice made?

Author response:
__Training on Positive__

We would like to clarify that this choice is standard for the IMDB positive text generation task. The original IMDB dataset does not have any reviews that start negative and finish positive. Thus when training an SFT model, it is standard to focus on the positive -> positive reviews as the task-relevant data for supervised learning. This was done in RL4LMs [1], Direct Preference Optimization [2], as well as popular RLHF codebases such as TRLx [3]. 

[1] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, Ramamurthy et al. 2023

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al. 2023

[3] https://github.com/CarperAI/trlx

Reviewer comment (Theory):
The motivation is not clear. Why guide policy should be integrated into RL finetuning especially in text generation? How the rollin & rollout scheme leads better text generation than PPO?

Author response:
The main motivation of this work is to improve on-policy reinforcement learning algorithms that have contributed to the success of ChatGPT. Despite their success, on-policy reinforcement learning algorithms can be very inefficient in exploration, and our goal is to address this issue. If we can address this issue then we can train policies that perform better. Our key observation, stated in our introduction, is that modern LLMs have impressive general language capabilities, which can help guide exploration for training models with reinforcement learning. Because there are numerous ways to provide guidance, we evaluate all combinations of guide paradigms (i.e., rollin-rollout interaction paradigms) to see which ones are better suited for text-generation.

The rollin-rollout scheme offers a paradigm to investigate various different possible ways that a guide policy can help with exploration. When rollin and rollout are with the learner policy then we reduce to normal PPO without any guidance. Rollin with a guide policy and rollout with a learner policy, means the guide policy provides more referent states where the learner policy can reset and perform optimization. These good reset states equate to the learner policy receiving high rewards much early during training. Seminal RL work like CPI [1] has shown that a good reset distribution can make RL to learn a higher quality policy. On the other hand, rollin with the learn policy and rollout with the guide policy means that the guide policy is providing feedback on the expected returns the learner partial generations will receive. Seminal imitation learning work like DAgger [2] shows that providing expert feedback on states generated by the learner is helpful for training a high quality policy. In this case, the guide policy is treated as an expert policy which the learner imitates and surpasses eventually. 

[1] Approximately Optimal Approximate Reinforcement Learning, Kakade and Langford 2002

[2] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Ross et al. 2011

Reviewer comment (Theory):
The theoretical justification section of the paper does not self-contained enough for readers.

Author response:
Thank you for this critique. Due to the space limit, we mainly provided intuitive explanations of the benefits of using a guide policy. Please see the answer above where we provided more connections to the seminal works in the RL and IL literature. If that’s not helpful, could we ask the reviewer to provide more actionable details on how we could make the section more self-contained?

Reviewer comment (Experiments):
Lack of ablation study for the mixing parameter $\beta$ which might be crucial in the framework.

Author response:
Thank you for this recommendation. We did not find this parameter to be particularly sensitive in our experiments, but we agree that this would be a valuable addition to the paper. We will work to add this to the appendix in the final version.

Reviewer comment (Experiments):
It would be interesting to observe the utilization of a more powerful guide policy, such as larger LMs like LLaMA, than the other policy in the framework.

Author response:
We definitely agree with the reviewer that this would be very exciting future work since in principle, the use of Llama2 as $\pi_g$ is perfectly valid. Note that given the growing concern of data contamination in LLM evaluations [1], experimental design becomes more difficult when evaluating on popular benchmarks such as IMDB and TL;DR summarization which have not been explicitly defined as held-out datasets by models such as Llama2 [1].

[1] Llama 2: Open Foundation and Fine-Tuned Chat Models, Touvron 2023

Reviewer comment (Reproducibility):
I am confusing on the setting $\beta = 1.0$ of AggreVaTeD in TL;DR. How $\pi_{\theta}$ of AggreVaTeD can be trained in this setting despite that it does not use $\pi_{\theta}$ at all?

Author response:
We would like to clear up a potential misunderstanding. When the mixing parameter is set to 1, we mean that all trajectories are mixed. That is every trajectory has some amount of $\pi_g$ tokens as well as $\pi_{\theta}$ tokens. For example in AggreVaTeD, this means that every trajectory has rollins from $\pi_{\theta}$ and rollouts from $\pi_g$. When doing the policy update, we would then update with the samples from $\pi_{\theta}$

Reviewer comment (Presentation):
Missing result for SFT + D2LOLS in TL;DR.

Author response:
Thank you for pointing this out. We will work to add these results to the appendix for the camera ready. We felt that investigating the individual components of D2LOLS (PPO++ and AggreVaTeD) was more valuable given our limited computational budget.

Reviewer comment (Evaluation):
Experimental results on both the sentiment sentence generation and TLDR dataset do not show that the proposed methods can outperform PPO, let alone significantly.

Author response:
Please see our updated results we posted along with the rebuttal for more results with TL;DR. For a quick summary, we show that when doing best-of-n sampling with our RL policies (PPO included), PPO++ shows greater performance improvement than PPO.

Reviewer comment (Reproducibility):
I am wondering what is the difference between the LLM policy \pi_\{theta} and the guide policy \pi_g. It is said that \pi_g is the SFT+nucleus sampling. But in PPO, the LLM policy model should also be a fine-tuned LLM on some tasks. In this case, what is the difference between the two policy models?

Author response:
Yes, you are correct. In our experiments, both our starting PPO policy $\pi_{\theta}$ and our guide policy $\pi_g$ are initialized with $\pi_{SFT}$. However, it's important to note that while $\pi_g$ remains fixed as a black box policy, providing PPO with additional states through the nucleus sampling decoding strategy, PPO itself is not fixed and gathers samples using the softmax sampling decoding strategy.

Reviewer comment (Experiments):
What is the size of the LLM policy model? GPT2-large or GPT2-medium? Have you tried larger LLM models?

Author response:
- IMDB: GPT2-base  
- CommonGen: T5-base  
- TL;DR: GPTJ (6B)  

We have the exact hugging-face model specifications in the implementation details in the appendix. Note that we aimed to try varying types and sizes of LLMs across our experimentation.

Reviewer comment (Evaluation):
What is the model for evaluating the output-perplexity?

Author response:
We computed Output Perplexity with GPT2-Large. We thank the reviewer for pointing this out and will make this clearer in the final version.

Reviewer comment (Experiments):
My main concern is the choice of baselines. While authors compare against SFT and basic PPO, prior research developed alternative algorithms for fine-tuning LLMs on human feedback that also claim superiority to PPO and SFT. Authors even cite some of those works in the paper. Some of those algorithms are: DPO[1], APA[2], P3O[3], SLiC-HF[4] though there may be more. These works adopt different means to learn from human feedback: some of them compatible to RLGF while others can only be used as competitors. I believe that the paper would be improved if, for each competitor, authors either compare against it in the experiments or prove that it has no chance of outperforming RLGF algorithms or, if authors claim that RLGF algorithms are orthogonal, demonstrate how it performs in combination with those approaches.

Author response:
Thank you for taking the time for a thorough review. We hope to address your concerns below:

__Choice of Baselines__

We agree with the reviewer that many of the cited works are relevant literature in the RLHF space. We’d like to emphasize two points: 
1. Our work focuses on investigating direct algorithmic improvements to PPO when using a reward signal
2. Our work extends beyond preference data for reward signals. For example, we tested on CommonGen which uses an automatic metric CiDer-D and SPICE as the reward function for the task. We aimed to show that given any reward signal, whether it be a preference reward model in RLHF tasks or an automatic metric such as CiDer-D, RLGF can better optimize than PPO.

We agree that it would be interesting to explore combinations of RLGF with many of the other methods; however, we believe this is well beyond the scope of a single paper, especially given the reviewer’s observation that this investigation is already quite dense.

Reviewer comment (Novelty):
For the last (and least) of issues, the idea of guiding the search for optimal policy is technically not novel outside the LLM fine-tuning domain, and the same can be said about other ideas presented in the paper. To reiterate, it is not a "problem" to be solved, and a good practical adaptation of existing methods is valuable in and of itself.

Author response:
Thank you for taking the time for a thorough review. We hope to address your concerns below:

__Paper Structure and Novelty__

Thank you for the suggestions. We can definitely see the merit of focusing on one or two algorithms in detail. On the flip side, we felt that demonstrating the full combination of guide-policy and policy interactions in the policy gradient formulation would be a valuable contribution to the space of Deep RL algorithms for text generation. As the reviewer pointed out, many of these ideas were explored in different domains, but we would like to emphasize that most of them did not have a deep algorithm counterpart with this work being the first to unify all of these methods into a single, policy gradient implementation. Furthermore, by investigating all these combinations, we hope we convinced the readers of which rollin-rollout interaction paradigms are better suited for text-generation than others.

On a separate note, many of the works cited were concurrent work to ours, with all of them being archival preprints or just announced Neurips acceptances at the time of submission.