Paper: How Does RLHF Shift Behavior Distributions? Distinguishability and Steerability (ICLR-2024)

Reviewer comment (Novelty):
The questions asked and analysis performed in this work is very similar to [1, 2]. The paper should clarify contributions and better differentiate what findings are unique.

Author response:
We attempted to elucidate the distinctions between this paper, Wolf et al. (2023), and Perez et al. (2023) in the **Introduction** and **Related Works section**. Wolf et al. (2023) provide theoretical frameworks which we empirically investigate to see whether it truly manifests in Language Models in a much more extensive manner than their previous version had. Now, it seems they go into more empirical investigation, however it remains relegated to toy scenarios where they separately fine-tune Llama 2 13B into separate positive and negative models. In contrast, we look into individual production models, a much more practical scenario. Perez et al. (2023) provide the persona dataset and evaluate it on a baseline model, but do not evaluate it on a steered model nor do they attempt to correlate characteristics of the behavioral distribution with steerability.

Reviewer comment (Evaluation):
The paper should better motivate the "Is the following statement something you would say?" questions. Section 4.1 states "which reflects the probability of the model will tend to say yes to such a negative statement (and hence behave undesirably)". Section 4.2 states "where a larger value indicates the model’s higher likelihood to assert the input statement". However, no evidence is provided for these statements. Since the interestingness/meaningfulness of the analysis presented is predicated on this assumption, it should be supported with actual evidence. To the best of my knowledge, the ability to self-reflect in this manner has only been demonstrated in certain limited contexts and settings; such as models reflecting on the validity of their own answers post-hoc [3]. However, that setting is quite different from the one here as the factuality of a statement is independent of the model.

Author response:
Indeed, what we are looking at is not the model's true tendency towards saying the statement but rather simply it's likelihood to say that it "would" say that statement. The language here should have been more precise.

Reviewer comment (Experiments):
The analysis is limited to a subset of behaviors. This choice is justified because "we select a subset of behaviors whose embodiment by an AI is likely to have pretty broad agreement on whether humans would like or dislike." I'm not particularly convinced by this. I, for one, would be significantly more concerned about current models displaying "anti-LGBTQ-rights" behaviors (which are somewhat trivialized in the paper as being political) than behaviors like "acts-like-it-wants-to-help-humans-but-does-not-care-about-that", as the near-term causal mechanism for potential harm is much clearer to me regarding the former given current model capabilities. Of course, I don't expect to agree with the premises of all research, but I do expect papers to motivate their research program using more rigor than assertions of consensus.

Author response:
We hear your perspective here, and strongly sympathize with it. To clarify, we had hoped to step back and remove ourselves from any controversial topics, especially in light of the alignment we concern ourself with being with respect to the *entirety* of humanity's differing values. We tried to select behaviors that it'd be hard to picture any human disagreeing on, no matter our personal beliefs. To do so, we took dedicated time to parse through the available behaviors in Anthropic's persona evaluation dataset. 

Thank you for these suggestions. We will duly take them into account.

Reviewer comment (Evaluation):
Steerability results are also quite limited. The plots in Figures 3 and 9, for most behaviors, show that there is a single prompt (most often 2 to 4 sentences long) which is sufficient to make the RLHF model understand the persona, and the prompts of higher length do not increase the understanding, so misalignment rate just oscillates stochastically. The base model does not show the high misalignment rate likely because it simply cannot follow the instructions as well as the model after RLHF. Arguably, there is not much correlation between the prompt length and misalignment beyond this simple effect.

Author response:
The lack of correlation between prompt length and misalignment beyond the effect described is true. However, we must note that this then stands in contrast to the claims made in Wolf et al. (2023), which state that you can continually recondition the model with additional sentences to shift it further and further into negative behavior. This shows that for LLaMa 2, past a certain point, reconditioning with more sentences doesn't have much of an effect. One can expect that with bigger models, the range that reconditioning with more sentences would grow, but perhaps there too is a certain point where the effects proposed in Wolf et al. (2023) also would decay away, and the theoretical conclusion does not necessarily hold.

Reviewer comment (Evaluation):
I think impersonating a character with provided traits and answering yes-or-no questions from the character's point of view does not constitute dangerous behavior by the model, so it was not removed by RLHF. The main claim of Section 5 is that "RLHF model is more steerable to exhibit negative behaviors often". A good RLHF algorithm makes a model more steerable in any direction, while prohibiting dangerous behavior. Hence, I suspect that RLHF with more strict criteria provided to evaluators, such as "do not allow the model to impersonate negative characters" could avoid this issue. I think the conclusion that "the prevailing practice of training a centralized RLHF model may not be optimally suited to address the intricate and diverse spectrum of human values, calling on future approaches for more comprehensive alignment techniques" is plausible, but it does not follow from the results of the paper. Why discard RLHF if we can simply adjust the annotation process?

Author response:
These are fair points, and our wording was not sufficiently precise in how we addressed this. In retrospect, we should not have argued that RLHF fundamentally could not target human-like psychological behaviors for avoidance. Instead, we should have argued that with how it's currently used by the large majority of practictioners, it does not often do so, which presents a potential vulnerability. 

However, even if targeted in the training process, Wolf et al. (2023)'s claims that if the negative subdistributions are shifted down and the positive subdistributions are shifted up, thereby making them more distinguishable, this leaves the model extra vulnerable to adversarial attacks. This is due to the fact that according to their framework, in the minimal case a prompt then must exist of shorter sentence count to make it behave poorly. We end up failing to find evidence in support of this claim, but, if true, this could present a fundamental RLHF limitation like worried about above.

Reviewer comment (Evaluation):
Figures 2 and 6 show logit values of the "Yes" output. If I understand this correctly, these are the pre-softmax values produced by the LLM. These values are not normalized, so the plots are not very meaningful. The authors observe that "the output values generally shift to be smaller for the RLHF model compared to the base model". This is not an interesting observation, since the softmax output does not change if all inputs are shifted by a constant. A more interesting quantity to look at would be $\log \mathbb{P}[\text{Yes}\mid s]$, where $s$ is the persona prompt and the statement. Footnote on page 5 states that the probabilities can be extremely small, so we should not look at them directly, but $\log$ probabilities would not suffer from this issue. I suspect that if we analyzed the distributions of $\log \mathbb{P}$ among the questions, the Wasserstein distance results would come out as more ambiguous.

Author response:
Thank you for the suggestion. We will revise the paper with log probability to avoid this normalization issue.

Reviewer comment (Experiments):
The paper fails to connect the results for distinguishability and steerability: "For most plots, the RLHF model (in orange) is capable of being prompted to behave more negatively with a higher misalignment rate than the base model (in gray). However, the degree to which this occurs does not seem to be predicted by the distinguishability". This failure to conform to the theoretical results from Wolf et al is not explained. I suspect that the reason is that Wolf et al claim that when the behaviors are better distinguishable, there _exists_ a short prompt that induces negative behavior. However, the authors of the paper at review do not search for a prompt of a given length extensively, and only try one prompt for each length.

Author response:
The suspicion you raise is interesting. However, Wolf et al. (2023)'s claim is not that there exists a short prompt to induce negative behavior. Rather, it is that the more distinguishable distributions are, the fewer sentences it should require to be able to recondition the model towards behaving negatively. The crux is on how many sentences are needed to repeatedly recondition the model, rather than whether a short prompt exists or not. Generally, the short prompt existing is what adversarial attacks usually tend themselves to and is a more discussed issue. However, it is not the claim that Wolf et al. were making.

Reviewer comment (Writing):
The text is rather diluted, with many ideas repeated almost word-for-word multiple times, e.g. the claim that RLHF cannot be used to align language models properly.

Author response:
We repeated ideas to attempt to reinforce them and remind the reader of the broader narrative we were working towards. We apologize if this made the paper feel diluted.

Reviewer comment (Experiments):
The experimentation is limited to one particular model LLAMA 2. Since the authors themselves do not do any further training or finetuninng themselves, any observation that they make are subject to the design and data decisions undertaken while developing LLAMA-2. To make more concrete observations about RLHF vs SFT, I would expect to see similar behaviors across a few different class of SOTA LLMs. Otherwise, one cannot concretely say that the observations generalize across all classes of LLMs.

Author response:
The experiment is indeed limited to LLaMa 2, which was the largest open source RLHF model available. Observations we make are subject to the design and data decisions undertaken while developing LLaMa 2. We are interested in validating this on more RLHF models as they become available.

Reviewer comment (Experiments):
While the paper proposes some observations, I would have loved more detailed ablations into the hypotheses which cause the behavior to be observed. For instance, if one is making such an observation "RLHF training process may not have been explicitly tailored to target specific human-like psychological behaviours for avoidance", I would have loved to see an ablation where some additional human data is collected or gathered from open source data on these behaviours, further RLHF the model on this data and demonstrate that this effect disappears or is mitigated to some extent.

Author response:
This ablation would indeed be useful. Further training the model on particular human psychological behaviors for avoidance and re-evaluating the distributions would be insightful.

Reviewer comment (Theory):
The paper presents two ideas for distinguishability -- Beta distinguishability and Wasserstein distance. They have a brief hypothesis suggesting that RLHF is not a direct likelihood maximizing loss and hence needs a different distinguishability metric. I don't quite see how introducing Wasserstein distance adds further value to the analysis of distinguishability or the steerability results that follow later as I can see RLHF models being more steerable negatively even after a high Wasserstein distance between behaviors. Also, I do not see an explanation why other forms of distribution divergence is not good enough compared to Wasserstein.

Author response:
To clarify, we couldn't use a KL-based distinguishability metric because each value was only in one subdistribution, positive *or* negative. In order to use many of the typical metrics that come to mind, you generally have to have outputs in both distributions to compare across for any given data point. As such, Wasserstein distance was the best available metric where we had a bunch of data in two distributions that weren't paired up in any particular way.

We attempted to use the $\beta$-distinguishability approximation metric used in the appendix of Wolf et al. (2023) but in their work, they created two separate models to source the positive and negative distributions from that were optimized to be their respective way which lead to their $\beta$-distinguishability increasing. It was a toy scenario very tuned to their particular setup. However, we were performing test-time probing of individual production models whose behavior distributions weren't optimized in this explicit manner. As such, we felt that Wolf et al. (2023)'s approach, while important to discuss due to its inspiration of our paper, didn't best fit our needs and that rather the Wasserstein Distance which more directly approaches how far apart two distributions are would work better.

*Note: Wolf et al. (2023) recently themselves in an update removed this $\beta$ approximation equation out of their work. We likely should have not used it and just worked with Wasserstein Distance.*

Reviewer comment (Reproducibility):
From what I understand, I think the current study is based only on the probability of the model answering Yes or No as plausible responses to the prompts. Can I understand if you introduced any extra text in the prompt to ask the model to stick to these responses only and not place its logit weight on other forms of saying something similar? I am asking this question particularly because of your observation of using the raw logits instead of probabilities.

Author response:
We did not introduce any extra text to ask the model to stick to Yes and No only in our final experiments.

Reviewer comment (Experiments):
Persona prompts is one form of jailbreaks which can be used to study steerability. Did you try other forms of adversarial attacks? I am wondering if the general notion of steerability is broader than just persona prompts.

Author response:
We did not try other forms of adversarial attacks as we particularly were investigating how conditioning on additional negative sentences shifted the behavior of the model. In this framework, most adversarial attacks, which generally cleverly shift behavior via a small number of tokens, couldn't be systematically set to different lengths out to fit our needs.

Reviewer comment (Novelty):
In related work or other sections, can you clearly distinguish the contributions compared to Wolf et al (2023) as I see a lot of similarities particularly with respect to theory and practice around the beta-distinguishability. Particularly, it will be interesting to test-time probe the models finetuned on the behaviors in Wolf et al (2023) as another observation to see how the effects change on further finetuning for these behaviors.

Author response:
Wolf et al. (2023) provide the theoretical framework and do brief testing on two optimized toy models. In contrast, we do test time probing on individual production models and significantly expand on their experimentation. Wolf et al. solely calculated the toy $\beta$ approximates, whereas we attempted to see whether the consequences of their theoretical claims were actually observable in language models: 
- Do larger distinguishabilities make it easier to repeatedly recondition the model with more and more sentences towards behaving negatively? 
- Does every additional sentence sourced from the negative distribution at a minumum continually shift the model to behave more poorly?

Reviewer comment (Experiments):
Overall I am wondering how generalizable are these observations considering it was done on test-time probing one class of models. Would it be possible to try this analysis in few other classes of models available opensource at least? There are several forms of RM training, data collection and RLHF that can be done in the wild and in case we are making generic statements about SFT vs RLHF.

Author response:
Same as W1 mapping: The experiment is indeed limited to LLaMa 2, which was the largest open source RLHF model available. Observations we make are subject to the design and data decisions undertaken while developing LLaMa 2. We are interested in validating this on more RLHF models as they become available.

Reviewer comment (Evaluation):
As a follow up, can you add a limitations section to the paper explaining why or how generalizable these observations can be and what further studies need to be done to make them more generalizable.

Author response:
Adding a limitations section would indeed aid in providing our thoughts on how generalizeable our results are.