Paper: A Long Way To Go: Investigating Length Correlations in RLHF (ICLR-2024)

Reviewer comment (Presentation):
It's quite difficult to parse what several of the abbreviations mean in the tables, and there is confusion about what exactly is shown in the first couple of figures and tables even after examining them for several minutes. It would be great to make the figures and tables more self-contained, for example, by defining abbreviations in the captions and summarizing takeaways, so the reader doesn’t have to keep bouncing between the figures/tables and the text.

Author response:
Thanks so much for this feedback, we uploaded a new version with figures and tables that we simplify and tie more clearly to presentation in the text, as well as clarifying experiments further to be easier to read.

Reviewer comment (Experiments):
Even if optimizing just for length, does the model still learn something nontrivial? A suggested point of comparison is an extra baseline where instead of sampling 8 outputs from the base LM and picking the longest one, one would sample a single output and forcibly continue generating past the end. This is expected to perform worse. This leads to the intuition that optimizing for length may also be optimizing for some less trivial notion of comprehensiveness.

Author response:
We thought that this was really interesting too, our hypothesis is that the joint optimization of length with the KL penalty is what leads to the “non-trivial” learning of features. Since repetitive, pathological outputs, etc. would likely have a higher KL divergence from the initial policy, this term likely forces the model to learn how to generate more descriptive outputs while ALSO maximizing length. This is probably why the lambda=0 baseline is sometimes much worse than the standard lambda=0.04 counterpart for pure length optimization.

Reviewer comment (Evaluation):
Confirmation is sought on whether weighted reward gain should be understood as measuring how much reward improves within a particular bin of length, i.e., the reward gain that comes from factors other than length.

Author response:
Yes, weighted reward gain gives an aggregate notion of reward improvement purely within bins

Reviewer comment (Evaluation):
There is curiosity about whether there is any qualitative sense of what the non-length axes are along which helpfulness improves in the experiments, i.e., any intuitions for the remaining 10-30% of helpfulness that wasn’t attributed to increased length.

Author response:
Thanks for asking! We thought a bit about this too. Based on looking at outputs and RM scores from before and after RLHF on some different settings, some alternate axes may be related to things like answer relatedness to the question (completely unrelated outputs will get lower scores, off-topic generation seems to go down from RLHF), and grammaticality. That said, a more comprehensive study of qualitative changes from RLHF (especially once we have more robust reward models in the future), would be interesting for future work.

Reviewer comment (Experiments):
As noted in the conclusion, it would be interesting to conduct a quick experiment on aspects such as harmlessness or honesty, even if only to confirm the intuition that length is primarily a heuristic for optimizing helpfulness and not really applicable to other common RLHF tasks. This is not necessary if it is troublesome to set up.

Author response:
We trained a reward model for harmlessness on the Anthropic data. We did not find length correlation in this model. The within-batch length correlation is around -0.3, and doing PPO with just the harmlessness reward model didn’t increase length either once converged. This is perhaps expected since a shorter response (such as abstention from answering) will often be harmless. We also observed that the outputs started to look strange eventually, so optimizing for just harmlessness has its own set of shallow features and should not be optimized on its own. Thanks for the suggestion!

Reviewer comment (Evaluation):
There is a lack of clarity on the concrete actionable suggestions for practitioners based on the analysis, although they are not strictly necessary, and it is acknowledged that several approaches to regularizing length were already tried.

Author response:
Thanks for asking! Some concrete suggestions are: (1) As you suggested, if one simply wants to get most of the “improvement” of RLHF without dramatic length increases, several of our proposed interventions are actually good options. (2) Downstream evaluation isn’t a gold standard for RLHF learning good features. Currently, better human/simulated preferences are often assumed to indicate that RLHF is learning “desired” features, dismissing further investigation. Especially if it’s high-risk (medical, legal application, etc.) RLHF practitioners should measure success in highly-specific terms, to validate assumptions about features learned. (3) RLHF work should focus mostly on reward modeling (better objective functions, collecting better data, engineering to enable larger scale with less compute, etc.). This direction is relatively underexplored, and will likely be the first step to substantial improvements

Reviewer comment (Evaluation):
The most confusing part for me is the evaluation. What is the rationale for adopting a length-bias metric, GPT-4 evaluation [1], when evaluating the correlation between length and RLHF performance? Two potential factors can be affected by length, making it hard to disentangle the attribution of length correlation. Or do you aim to reveal the bias issue of GPT-4 in this paper? Then could you please explain more clearly what you mean by "length correlations in RLHF" and what the length correlates with? The reward modeling? The optimization algorithm? The evaluation? Or all of them? If so, I'd suggest investigating them separately.

Author response:
> Could you please explain more clearly what you mean by "length correlations in RLHF" and what the length correlates with? The reward modeling? The optimization algorithm? The evaluation? Or all of them? If so, I'd suggest investigating them separately.

We examine “length correlations” in this paper in the following contexts: 
- Preference data: Is label distribution imbalanced to favor length?
- Reward modeling (Section 4): Does reward score correlate with length? Does “easiness” of examples correlation with length? (connects to preference data)
- PPO (Section 3.5): Does PPO lead to longer outputs if we do things like increase the KL penalty?
Since these are different notions, we generally dedicate different sections / experiments to each of these components, but we can make sure to clarify these distinctions further. Importantly, we *don’t* seek to analyze length bias within evaluation, since our hypotheses largely don’t depend on GPT-4 evaluation. 

> The most confusing part for me is the evaluation. What is the rationale for adopting a length-bias metric, GPT-4 evaluation [1], when evaluating the correlation between length and RLHF performance? Two potential factors can be affected by length, making it hard to disentangle the attribution of length correlation. Or do you aim to reveal the bias issue of GPT-4 in this paper? 

Yes, both GPT-4 and human evaluators likely have a bias towards longer responses (recall our goal isn’t to reveal these evaluation biases), but we find they can still reveal useful information when length biases are controlled (see claims below). For comparability with prior work, we follow the protocol of [3] for “simulated preference” downstream evaluation, as it was shown to correspond well with human preference.

We use simulated preferences as follows: 
- Context: RLHF generally makes “numbers go up” relative to SFT on standard settings (Table 2), which largely follows prior work
- Claim: Reward interventions can improve quality of outputs independent of length (Table 5)
- Claim: Length-only PPO is both better than a baseline of sampling longer outputs, and comparable to standard PPO (Table 6). 

Importantly, notice that for both claims, the simulated win rates are actually higher for models with much shorter lengths on average: these results are not explained away by saying that GPT-4 prefers longer outputs in simulated win rate experiments.

Finally, note that most of our results (Fig 2,3,4; Table 1,2,3,4) are based on intrinsic measures like reward model scores and output length, which are unaffected by any biases from GPT-4.

Reviewer comment (Experiments):
I am skeptical about the experiment and claim that "reward models are not able to make progress on most training examples" (sec 4.1). The results may, at least partly, be due to the reward model capacity. I suppose that reward modeling is a relatively difficult task, so weak models may only capture some shallow features. With a more capable reward model, the "hard" samples may be learned. A supporting evidence is that, as shown in Table 3, the length heuristics accuracy on WebGPT is 55.7%. However, in the result of a recent paper [2], a more powerful reward model that is not trained on WebGPT can yield 65.2% accuracy. Therefore, according to this paper, at least 9.5% of counter-biased samples are learned by the reward model. Perhaps the authors can supplement experiments by adopting more powerful reward models, e.g. scaling the base from LLaMA-7B to 70B following the original LLaMA-2 paper [3] (if computation allowed), and see if the learning patterns remain the same.

Author response:
> I am skeptical about the experiment and claim that "reward models are not able to make progress on most training examples" (sec 4.1). The results may, at least partly, be due to the reward model capacity. [...] Perhaps the authors can supplement experiments by adopting more powerful reward models [...]

Thanks for mentioning this! We actually agree with this point. We didn’t intend to claim that reward models *in general* couldn’t learn from most examples on those datasets, but rather that this happens with the 7B scale reward models with current objectives and data, then propagating length correlations throughout RLHF. This is why a big takeaway for future work is to focus on improving reward modeling especially at more scientifically feasible scales. 

We did our experiments with 7B-sized models largely due to hardware limitations (because PPO involves storing several large models). Note that:  
- The majority of open RLHF work uses 7B, or often even smaller-scale models [1][2][3][4].
- *All* our trained 7B reward models actually outperform length heuristic (61.5% on WebGPT vs 55.7), and thus are likely learning other features, yet PPO still largely optimizes for length.
- Regarding UltraFeedback (the paper with 65.2% accuracy on WebGPT, which we note was arxived since the ICLR deadline), scale may be a component, but they also use very different (potentially better) preference data. This supports our overall message of needing to improve preference data and reward modeling

That said, based on your feedback, we scaled our RM experiments up to LLaMA-2 13B. We report intrinsic reward modeling accuracy results below (compare with Table 4 in the paper):  

| Model | WebGPT | Stack | RLCD |
|-------|--------|-------|------|
| LLaMA 7B    | 61.5%  | 70%   | 80%  |
| LLaMA-2 13B   | 64.5%  | 71.3% | 81.2%|

As shown in the table, we found that reward modeling accuracy is only marginally better than it was before, suggesting that model scale likely isn’t the main bottleneck on the “hard” examples. We agree it would be good to scale up these results further, but we think the main takeaway of our results and those in UltraFeedback is the need for better datasets rather than just scaling. 

Thanks again for the experiment suggestion!

Reviewer comment (Novelty):
Although the experiments are well-executed and do support the main message, this pattern has been observed in numerous other studies, and strategies to address this bias/reward hacking have been extensively documented elsewhere. Consequently, the contribution of this paper does not look very strong for an ICLR conference.

Author response:
We provide the most in-depth study of this phenomenon to date, indicating that length may play a much larger role than previously anticipated by the community. Length increases from RLHF have been observed previously (e.g., in “Secrets of PPO”, which we cite), but we believe our analysis, and the new tools we introduce, contribute a lot to our understanding beyond just observing the pattern. 

For instance, we conclude the following, which are not documented in prior work:
- PPO’s reward optimization strategy relies *substantially* on length at the cost of other features
- PPO’s overall reward optimization strategy is qualitatively similar across various RL interventions given the same reward model: RMs are the most crucial component
- Reward robustness issues are the primary source of length biases; current open reward models struggle on preference data, and *heavily* rely on “easy” subsets of training data with shallower features
- Doing PPO with *just length* is comparable to full RLHF with trained reward models

If there are additional papers we should cite or contextualize our work with respect to, we’d love any pointers!

Reviewer comment (Presentation):
The "STD" term was introduced in Figure 2 way before defining it later in the text.

Author response:
Thanks for pointing these out! We made a further pass through the paper to clarify things, especially key take-aways from experiments, and improve captions / Figure 3 based on your feedback.

Reviewer comment (Presentation):
For Figure 3, why not have both settings (red and black) for all the lengths? Does that mean that sometimes there are no examples in a particular bin for a particular setting or are there some missing datasets in the figure?

Author response:
Yes. Due to length shifts, there are cases where, for example, PPO doesn’t generate any outputs at shorter lengths, leading to us being unable to have a comparative bin (which is one of the reasons why we use a length-constrained high-KL version to better validate our hypotheses in a case with more overlapping bins). 

We realized that putting both versions in the plot was a bit overloaded, so we made a simpler version with only the high-KL arrows, keeping both normal and high-KL numbers in the WRG table and moving the 2-color plot to the appendix. Thanks for the feedback!