Paper: Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR-2024)

Reviewer comment (Novelty):
Unclear method contribution: the proposed method is nearly, if not exactly, a special case of TRPO/PPO method in the bandit setting. Such a special bandit instantiation has been widely considered in classical RLHF works, such as [1,2].

Author response:
We believe this comparison is an oversimplification. PPO/TRPO are online RL that assume the training trajectories are generated by constant interaction with the environment. Our method is offline and assumes access to only static data. Accordingly, value estimate, advantage, and the training process differs significantly from the online methods.  
Thank you for references to PPO in single action bandit setting; we will update Table 1 and the intro. Most modern implementations of PPO for LMs [12,13,14] use per-token action assumption with trajectory level sparse reward (only the last token gets reward). This makes the value estimation of intermediate tokens difficult. Instead of on-policy updates, we use a sequence-level value estimate of reference policy that we cheaply train from its validation performance. To our knowledge, such an easy-to-compute sequence-level value estimate has not been used before for offline RL training of LMs. We also show a combination of algorithmic modifications to improve training efficiency and performance (point 6).

Reviewer comment (Novelty):
Advantage-weighted policy optimization is also a well-studied method in offline RL. For example, Eq. 5 in this paper is very similar to Eq. 4 in AWAC [3], except for the importance weighting that basically comes from TRPO/PPO.

Author response:
While it is true that Advantage-weighted Regression [3] is a well-studied method in offline RL, there are other differences from A-LoL. AWR maintains two identical models for value estimate and policy and it trains both models on samples from D. Here, D can collect additional off-policy trajectories. In contrast, A-LoL estimates sequence-level value on top of a frozen LM’s validation performance only once before target policy training. This allows A-LoL to discover a subset of positive advantages within the static training data, since negative advantage instances are detrimental to performance (point 6).

Reviewer comment (Novelty):
Weighted behavior cloning has been quite extensively used in prior NLP papers, e.g., [6,7,8,9,10,11]. It will make the algorithmic contribution of this paper more clear if the authors can have a paragraph discussing and comparing with such related works, instead of only citing the CRR paper from offline RL.

Author response:
Thank you for sharing the related works! We use weighted behavior cloning as a reward-based offline RL baseline and will add a paragraph discussing these additional related works.

Reviewer comment (Evaluation):
In Table 2, the comparison with PPO may not be fair, because the reward for PPO is a good-or-bad classifier. The offline RL methods, on the other hand, are fitted towards the original responses that themselves show high linguistic diversity, which would implicitly guide the offline RL methods, especially A-LoL, towards generating longer and more diverse sequences. In short, there is no guiding signal for PPO to generate such sequences, while the offline RL methods implicitly have the guidance.

Author response:
Common RLHF benchmarks and PPO implementations [12, 13, 14] use sequence-level preference classifiers as reward models. For a fair comparison, we keep the same initial reference LM and reuse data, and reward models [21] from PRO’s [20] experimental setup (a preference-based offline RL baseline). In the PPO baseline, we use nucleus sampling (top-p=0.95) and also include entropy in the loss term to encourage diversity.  
All offline RL baselines use the same data, but vary in diversity. In particular, DPO [18] loses diversity, whereas all reward-based baselines and A-LoL methods show better diversity.

Reviewer comment (Experiments):
There are several well-established exogenous components in the proposed method, such as (1) importance clipping, (2) discarding negative advantage datapoints, (3) prioritized sampling. It is unclear how each of those exogenous components contribute to the overall performance. It is also unclear if the baselines can also benefit from such exogenous components, e.g., (2) and (3). This again muds the algorithmic contribution of the proposed method and make the experiment results less convincing.

Author response:
Thank you for pointing this out. We compare the impact (1) and (3) in the ablations in Appendix B.3 (Figure 6). We find (1) importance clipping is crucial for A-LoL; it degenerates to NaN losses without it. We find (3) priority sampling is notably better than random sampling. We added a new ablation for (2) and found that negative advantage is detrimental. 
For a fair comparison with reward-based baselines, we employed reward-based priority sampling (page 4), but do not discard the negative advantage data as they do not have an internal value estimate like A-LoL.

Reviewer comment (Theory):
For the “ref free” variant: Is it justified to use 1 as importance weights, in the reference free variant of the algorithm? I can’t wrap my head around whether that’s an acceptable approximation, or whether that's making the derived Equation (3) or Equation (4) simply incorrect.

Author response:
Yes. A-LoL reference free is effectively the supervised LM advantage multiplied with negative log-likelihood loss. Equation-wise, it is very similar to the weighted behavior cloning baseline which is the reward multiplied with the log-likelihood. Another way to look at reference-free is to use the clipping parameter $\epsilon = 0$. A-LoL reference free should be viewed as an ablation as we mainly compare with it to investigate the impact of importance weight.

Reviewer comment (Theory):
For the “sequence” variant: The approximation of importance rule is a bit strange. See line 6 of the “variants with alternative importance weight” paragraph on page 4. It’s essentially saying a1 * a2 * … * aT * (b1 + b2 + … + bT) = a1 * b1 + a2 * b2 + … + aT * bT. But this seems wrong? Am I understanding this correctly? Perhaps an explanation of why this is a good approximation will be helpful. But at the same time, the empirical results aren’t really impacted much, so I’m conflicted on how much I should treat this approximation seriously.

Author response:
To clarify, the sequence variant importance weight is an approximation and not mathematically equal. Intuitively, it tries to reweight each token’s likelihood with its token importance weight while still using trajectory-level advantage values. Empirically, we find this approximation to improve diversity in all the tasks while also showing better training convergence than other A-LoL variants. We also find quantitative differences in how the sequence variant affects the importance weights (more in answer to question 2 below).

Reviewer comment (Novelty):
Phrasing of the main question in paragraph 1 – the main question seems to be the italicized sentence at the end of the first paragraph: “can we perform rewarded learning, similar to PPO, while only using pre-existing data” but the answer is already yes given the literature in the past few years.

Author response:
We thank you for sharing more related work. The main distinction with previous works is that A-LoL doesn’t need model-generated data during target policy training and can use arbitrary classifiers as rewards. Both cringe loss [1] and unlikelihood training [2] require contrastive trajectories from the learner model during its training loop. Director [3] needs to train the classifier alongside the target LM and cannot use arbitrary pre-existing classifiers as rewards. By “similar to PPO”, we mean the ability of PPO to incorporate arbitrary non-differentiable rewards while using pre-existing language data. We will update the text to highlight these distinctions.

Reviewer comment (Novelty):
Cringe loss (https://aclanthology.org/2023.acl-long.493.pdf), as well as the older director loss (https://arxiv.org/pdf/2206.07694.pdf) and unlikelihood loss are relevant. The other algorithms the authors cited are also examples where we can learn from rewards while only using pre-existing data. I think the authors’ research question can be more specific & take prior work into account.

Author response:
We thank you for sharing more related work. The main distinction with previous works is that A-LoL doesn’t need model-generated data during target policy training and can use arbitrary classifiers as rewards. Both cringe loss [1] and unlikelihood training [2] require contrastive trajectories from the learner model during its training loop. Director [3] needs to train the classifier alongside the target LM and cannot use arbitrary pre-existing classifiers as rewards. By “similar to PPO”, we mean the ability of PPO to incorporate arbitrary non-differentiable rewards while using pre-existing language data. We will update the text to highlight these distinctions.

Reviewer comment (Theory):
Given that in language, the transition function is trivial - it is unclear why offline algorithms are more sample-efficient than online algorithms. Offline algorithms assume access to a lot of quality data, while online algorithms can work with small amounts of data and well-designed reward functions.

Author response:
This is not always the case. Online RL is only sample efficient if exploration yields high-quality data, and the reward function is well-designed. Exploration with LMs is tricky due to exponentially large output space [2,3] and reward arrives at the end of the output [4]. Instead, offline RL operates on static training data. A-LoL can further filter low-quality instances using negative advantage. In the HHA task we allocate a similar number of gradient updates to all baselines and PPO shows slower growth (Fig 3) compared to offline RL methods (Fig 2).

Reviewer comment (Theory):
The paper relies on the assumption that each token is not an action but instead a sequence is an action, but it is unclear why this assumption matters. Most RLHF techniques optimize policies on sequence-level losses, not token-level losses because the reward functions are defined on a sequence.

Author response:
Most prominent implementations of PPO in RLHF use per-token as action and assign the full sequence reward to the last token [5,6,7]. This makes the value estimate of intermediate actions tricky and also affects the loss values at each token. Thus, concurrent works are investigating entire output as single-action in offline RL [1,8] for LMs. Using the single-action assumption, our method trains the reference LM value-estimate of the entire sequence and freeze it before training target policy, which allows for stable training (Fig 2).

Reviewer comment (Theory):
For the A-LOL algorithms to work, there is a set of assumptions that are not explicitly mentioned in the paper that could have a big impact on performance. In particular, if you don't have good data coverage and a good initial policy, then A-LOL will fail, which means that A-LOL and PPO in RLHF have the same assumptions.

Author response:
Thank you for highlighting this point. We agree that a bad initial policy or lack of good data coverage can be detrimental to the performance of A-LoL. We will clarify this in the limitations section (Appendix D).

Reviewer comment (Experiments):
The experiments did not include PPO due to a seed collapsing, but there has been evidence in the literature that this does not happen, which means the authors did not tune this baseline algorithm properly [1].

Author response:
We include the average PPO performance for the two out of three seeds (Table 2). We followed the experiment setup from the PRO baseline [10] which used a 1.4B parameter reward model [11] for training and a 6.9B parameter model for evaluation. The reward model used in the provided reference [1] is a 6B parameter and even the reference policy is different from what we used in our experiment, which could explain the difference in PPO’s performance.  
For hyperparameter tuning, we selected one seed and tested with PPO rollout batches in {16, 128} and learning rates in {2e-4, 2e-5, 5e-6, 5e-7} and found the best configuration be batch size=16 and lr=2e-5. Extending these hyperparameters to 3 seeds resulted in one of the seeds collapsing which highlights the instability in on-policy RLHF [1,2,3]. To verify the correctness of our PPO implementation, we also include the PPO checkpoint from an [external source](https://huggingface.co/reciprocate/ppo_hh_pythia-6B), but found our PPO implementation to work better.

Reviewer comment (Experiments):
The authors claim that their proposed approach is more data-efficient because they filtered out 33% of good responses, but the same procedure can be done for other techniques. However, a similar procedure was not conducted for the baseline algorithms to show that with less data, their proposed approach performs better.

Author response:
The reward-based and preference-based offline RL baselines do not come with any principled method to filter data on their own. To filter data with rewards, one would need to know the reward landscape of each training dataset and also analyze the low reward instances to pick appropriate thresholds [9], which is out of the scope of this study. A-LoL is unique in its ability to automatically detect the subset of positive advantage in any training set using the sequence-level value estimate of the initial supervised LM. The amount of negative advantage varies for each task, initial LM, and data split, for example, 33% of the response in HHA task (Section 4), 48% of the downvoted, and 36% of upvoted reddit comments (Section 5) and 10%, 39% and 55% of different data splits of knowledge-grounded dialog (Appendix C.2).