Paper: RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment (ICLR-2024)

Reviewer comment (Experiments):
As shown in Appendix C, RLAIF-Few-30B produces both a better preference model and a better-aligned language model than RLCD-30B on the harmlessness benchmark, which is attributed to few-shot prompting by the authors. It seems that this technique can also be integrated into RLCD to enable a fairer comparison.

Author response:
We intentionally used zero-shot prompts in our main experiments to minimize the influence of prompt selection for fair comparison (more detailed discussion under Reviewer HiRS, Concern 1); adding few-shot examples would introduce an additional variable. We mainly included RLAIF-Few-30B in Appendix C to check that we can replicate the results from [2], to make sure that RLAIF’s often poor performance when we do not use both (1) few-shot and (2) 30B scale is not due to implementation differences.

To be clear, we think the final language model learned by RLAIF-Few-30B is not good - as shown in Table 17 in Appendix C, the model distribution collapses to a generic but unhelpful output, with *nearly all* outputs being a variation of “I'm sorry, but I'm not sure how I can help with that. Can I ask some questions to help me understand your problem better?”. Quantitatively, the fraction of distinct 3-grams (Dist-3) within a sample of 10000 tokens is only 18% (!) for RLAIF-Few-30B, compared to over 90% for most other experiments (see new Appendix J.3; we’ve also added more discussion on the mode collapse to Appendix C).

In any case, we’ve now run RLCD-Few-30B on harmlessness too. RLCD-Few-30B also collapses to a single generic output: “I'm here to help you get the information you need. Can I help you today?”. GPT-4 happens to prefer RLAIF-Few-30B’s generic output, but we think that it’s more indicative to look at the preference model’s performance since both models are so heavily mode-collapsed. RLCD-Few-30B’s preference model’s accuracy for agreeing with humans is 62.8% (avg. prob of agreement = 0.599), a good bit higher than the 57.0% for RLAIF-Few-30B’s (avg. prob of agreement = 0.540).

Reviewer comment (Novelty):
The proposed method is too simple that seems like a prompting trick in preference data construction.

Author response:
The core idea of RLCD is to change the generation process of the preference outputs, moving away from noisy near-boundary examples to more reliable far-from-boundary examples; this tradeoff is both empirically effective and automatable. Please see “Additional Aspects of Our Contribution” under the General Response for a more detailed discussion of our contributions; we would also like to highlight our theoretical motivation for RLCD in the new Appendix N, which we summarize in “Theoretical Motivation for RLCD” in the General Response.

RLCD then relies on prompting because—given that we want to change the generation process of the preference outputs—prompting seems to be the simplest implementation of our idea. We view this simplicity of implementation as a strength which makes RLCD easier to reuse or adapt in new applications; we note that reviewers HiRS, UFW8, and RAFU mentioned simplicity / straightforwardness of implementation of RLCD under Strengths as well.

Reviewer comment (Experiments):
More comprehensive methodology and solid experiments (e.g., stronger baselines and deeper analyses) are needed to improve the contribution and soundness of this paper.

Author response:
We’ve now added experiments according to your questions. Was there something else in particular you were looking for?

Reviewer comment (Evaluation):
Will the proposed method reduce the diversity of preference data?

Author response:
We’ve now checked the preference data diversity by measuring the percentage of distinct 1-, 2-, and 3-grams among 10K tokens of preference data for both RLAIF and RLCD (see new Appendix K.2). The metrics for RLCD’s preference data are overall very similar to those of RLAIF; if anything RLCD has higher diversity more often than not.

Reviewer comment (Experiments):
What if the original prompt already contains positive or negative instructions?

Author response:
We think adding positive or negative instructions to the original prompt is fundamentally orthogonal to the RLCD idea, which is mainly about creating *contrast* in the outputs $o_+$ and $o_-$ by using *contrasting* prompts $p_+$ and $p_-$. To confirm, we’ve also added an experiment in the new Appendix L, where we observe that adding positive instructions to the prompt doesn’t change RLAIF’s performance compared to RLCD.

Reviewer comment (Novelty):
The contribution of the paper is limited.

Author response:
Please see “Additional Aspects of Our Contribution” under the General Response.

Reviewer comment (Evaluation):
In the GPT-4 evaluation results (Table 3), the proposed RLCD algorithm gains a bigger advantage on the smaller LLaMA-7B model when compared with its baselines. Why does this occur? Is it due to the poor performance of RLAIF with smaller models?

Author response:
Indeed, RLAIF works quite poorly with smaller models, which struggle to label preference data accurately using the RLAIF scoring prompts (see Tables 24 and 25 for examples, if interested). Please see our “Theoretical Motivation for RLCD” under the General Response, which includes potential explanations for why RLCD works so much better on 7B, or the new Appendix N for complete details on these theoretical intuitions.

Reviewer comment (Evaluation):
The human evaluation always preferred RLCD compared with RLAIF, in contrast to GPT-4's preference on RLCD-30B. Can the authors please provide some insights into the cause of the difference in the GPT-4 vs. human evaluation?

Author response:
We discussed the annotation disagreements briefly in Appendix G, where we observed quantitatively that GPT-4’s agreement with humans is lower in two main cases. 

First, when evaluating helpfulness on the harmlessness prompt set, GPT-4’s own harmlessness alignment often prevents it from saying which answer is more helpful. 

Second, specifically when comparing RLCD-30B to RLAIF-30B, the disagreement is higher because both models are actually producing quite good outputs (see e.g., example helpfulness outputs for RLCD-30B and RLAIF-30B in Table 27), so the comparison may simply be more subjective. For example, with outlines, we noticed qualitatively that RLCD’s outputs seemed a bit more surprising, which was fine for humans but somewhat dispreferred by GPT-4. So we think GPT-4 likely just subjectively prefers a particular style, and that it’s a bit more reliable to survey humans as a result.

Reviewer comment (Reproducibility):
The downstream fine-tuning is using PPO. Although not the main contribution of the paper, more details of the downstream fine-tuning procedures are requested, such as the inputs and outputs of the RL model, the reward used by PPO derived from the upstream preference generation, and the overall process of the RL fine-tuning. This information would be useful for a general audience who would like to use the proposed RLCD method.

Author response:
Good point, we previously included the implementation details and hyperparameters, but we’ve now added a description of the actual PPO fine-tuning procedure in Section 3.1.1 in the main text.

Reviewer comment (Theory):
The method seems straightforward and is less technically strong. The main technical contribution is the simple changes on the prompt design when generating the pair of responses for the preference data. While such changes make sense intuitively, i.e. making both responses more contrastive, the authors didn't show much principled analysis on why such design can be better than direct RLAIF. I think it would be helpful to give some more technical/principled explanation on RLCD.

Author response:
Thanks for this excellent suggestion! We’ve now added a technical explanation for our intuitions; please see “Theoretical Motivation for RLCD” under the General Response, and the new Appendix N for complete details.

Reviewer comment (Experiments):
I wonder if this design is important when there's human feedback data presented in the preference dataset as well, which is more common in practice. It would be interesting to see if RLCD would still make such a big difference in practice with some human feedback data in the mix.

Author response:
Since we have the original Anthropic human-labeled preference data for harmlessness and helpfulness, we’ve now tested harmlessness and helpfulness on 7B model scale with human preference labels mixed in (see new Appendix M). Even with as many as 20% human labels, RLCD still outperforms RLAIF by a decent margin according to GPT-4, though the difference is naturally smaller than with no human labels.

Reviewer comment (Experiments):
Please show some experiments in scenarios where there's human preference data available in the data mixture.

Author response:
Please see response to Concern 2.

Reviewer comment (Experiments):
It is hard to understand if the performance is coming from the prompts themselves or the proposed algorithm.

Author response:
> **Concern 1: It is hard to understand if the performance is coming from the prompts themselves or the proposed algorithm.**

All RLAIF scoring prompts and RLCD affix pairs are shown in Appendix A, together with a description of how we selected them to maintain fairness of comparison with our baselines. 

RLAIF uses “discriminative” prompts for scoring while RLCD uses the prompt affix pairs in generation, so there are necessarily some differences where the language is unnatural when converting directly from one to the other, but we attempted to match the general meaning. For example, for harmlessness, we used the original 16 scoring prompts from [1] for RLAIF, and then constructed 16 corresponding affix pairs for RLCD to paraphrase the qualities being evaluated in those scoring prompts. Due to ensembling 16 prompts, we think that our harmlessness experiments should be particularly robust to the effect of prompt selection; the results also hold up when we run a version of harmlessness with a different set of more “focused” prompts in Appendix D. For helpfulness and outlining we wrote our own scoring prompts for RLAIF, and each scoring prompt directly corresponds to an affix pair designed to encourage differences on the same axis being measured.

Meanwhile, context distillation directly uses the same positive prompt $p_+$ as RLCD, to minimize the effect of prompt selection as much as possible.

Reviewer comment (Experiments):
The authors do not provide a thorough description of the outlining prompts task, and there does not seem to be any references for this task, so it is very hard to understand the task's difficulty.

Author response:
> **Concern 3: The authors do not provide a thorough description of the outlining prompts task, and there does not seem to be any references for this task, so it is very hard to understand the task's difficulty.**

We were intending to provide a more thorough description along with examples, but regrettably were blocked due to data licensing restrictions. We’re revisiting to see if we can release more details, though in any case we consider the harmlessness and helpfulness tasks to be the more “standard” tasks for our main experiments. We’ll still add some more information on outlining where possible—e.g., the premises are usually 10 to 40 tokens, to get a sense of length. The outputs in our main experiments were anywhere from 20 to 200 tokens or more depending on the method.

Reviewer comment (Experiments):
How did you decide on the prompt affix pairs?

Author response:
> **Question 1: How did you decide on the prompt affix pairs?**

Please see description under Concern 1.

Reviewer comment (Experiments):
Why is having more than one prompt affix pair important?

Author response:
> **Question 2: Why is having more than one prompt affix pair important?**
	
Having more than one pair isn’t required, actually—in our helpfulness task we only use a single prompt affix pair. But we think using multiple affix pairs can be convenient when trying to optimize more complex multi-objective tasks such as outlining, where we are trying to optimize for interestingness as well as well-formedness and premise relevance (hence, we use 3 affix pairs). [1] also used 16 scoring prompts for harmlessness in RLAIF to increase robustness to prompt design, which we matched in our work.

Reviewer comment (Evaluation):
Given that you automatically assume $o_{+}$ is preferred - how often does $o_{+}$ have a lower reward with respect to a held-out reward function?

Author response:
> **Question 3: Given that you automatically assume $o_+$ is preferred - how often does $o_+$ have a lower reward with respect to a held-out reward function?**

We’ve now run this analysis for harmlessness and helpfulness using the same held-out reward models described under Concern 2; see new Appendix K.1. It’s still noisy as we’re relying on the base pretrained LLaMA: the fraction of correct labels w.r.t. the held-out reward ranges from 54% (harmlessness, 7B) to 74% (helpfulness, 30B). However, RLCD’s label accuracy is always higher compared to RLAIF, with the difference ranging from 8% to 14% absolute.

Reviewer comment (Theory):
Why would training examples far away from the boundary be better than training examples close to the boundary? I would assume that the points far away could be easy to classify.

Author response:
> **Question 4: Why would training examples far away from the boundary be better than training examples close to the boundary? I would assume that the points far away could be easy to classify.**

If we could guarantee that the training examples were correctly labeled, then yes we would prefer examples close to the boundary. However, RLAIF/RLCD need to use a model to label the training examples in the first place before they can be used downstream. RLCD’s examples may on average be farther from the boundary, but are more likely to be accurately labeled. In fact, RLCD’s label accuracy may actually be higher *even when considering only examples that are close to the boundary,* as suggested by our theoretical analysis in the new Appendix N; please see “Theoretical Motivation for RLCD” under the General Response.

Reviewer comment (Reproducibility):
Could you elaborate on how you performed your GPT-4 evaluation? What prompt did you use? How did you shuffle the data? Etc?

Author response:
> **Question 5: Could you elaborate on how you performed your GPT-4 evaluation? What prompt did you use? How did you shuffle the data? Etc?**

The procedure is described in Appendix F.2; the full prompts are shown in Tables 21 and 22. When comparing two outputs, we show them in random order. (Please let us know if you meant something else regarding data shuffling.)

Reviewer comment (Evaluation):
Did you perform GPT-4 evaluation using comparisons from the algorithms-generated output and human-generated data for a given prompt?

Author response:
> **Question 7: Did you perform GPT-4 evaluation using comparisons from the algorithms-generated output and human-generated data for a given prompt?**

Our GPT-4 pairwise evaluation compares algorithm-generated outputs from two different methods (e.g., RLCD vs. a baseline) on the same prompt. There are no human-generated outputs being compared. (Note we don’t have human-generated *outputs* in the Anthropic data we’re using, only human *preference labels* on model-generated outputs.)

Reviewer comment (Evaluation):
Could you provide other diversity metrics on the outputs of the text generated? (e.g., the ratio of distinct n-grams (Distinct-1, Distinct-2), average length of sentences, or count of n-grams in the generated text [1]). The text in Table 4 implies that RLCD generations are much longer than the base model sentences.

Author response:
> **Question 8: Could you provide other diversity metrics on the outputs of the text generated? (e.g., the ratio of distinct n-grams (Distinct-1, Distinct-2), average length of sentences, or count of n-grams in the generated text [1]). The text in Table 4 implies that RLCD generations are much longer than the base model sentences.**

Good idea, we’ve now checked these. 

The n-gram diversity for RLCD is very similar to that of baselines (new Appendix J.3), except for on harmlessness on 7B scale only, where RLCD’s diversity is lower due to repetitive wording when refusing to answer some requests (though it’s still far from completely mode-collapsed, as shown in examples in Table 26). 

As for length (new Appendix J.4), RLCD generations are indeed often longer than those of baselines particularly for helpfulness and story outlining, as RLCD correctly identifies that longer outputs better satisfy these alignment criteria on average (as you noticed in the Table 4 example for the helpfulness task).

Reviewer comment (Experiments):
For RLAIF, did you run an experiment where you sample two outputs from the $p_{+}$ positive affix prompts? This provides RLAIF algorithms with modified prompts similar to RLCD and would reduce the advantage that RLCD has to strictly have the altered $p_{-}$ prompts.

Author response:
> **Question 10: For RLAIF, did you run an experiment where you sample two outputs from the $p_+$ positive affix prompts? This provides RLAIF algorithms with modified prompts similar to RLCD and would reduce the advantage that RLCD has to strictly have the altered $p_−$ prompts.**

The core intuition of RLCD is to use the different prompts $p_+$ and $p_-$ to amplify the *contrast* between the outputs $o_+$ and $o_-$. In principle, $p_+$ isn’t even needed for contrast—one could imagine a version of RLCD that uses just the base prompt $p$ with a negative prompt $p_-$ to produce $o$ and $o_-$ for contrast instead. So we think that just directly sampling two outputs from the same $p_+$ wouldn’t really change the advantage of RLCD over RLAIF at all, which we confirmed in 7B experiments in the new Appendix L.