Paper: Human Feedback is not Gold Standard (ICLR-2024)

Reviewer comment (Evaluation):
I’m not sure I’m convinced that “Human evaluation is necessary” (as stated in the conclusion). Techniques such as RL from AI feedback have been shown to be as effective as RLHF for increasing the harmlessness of LMs [2]. They ultimately rely on humans, but these are human experts involved in prompt engineering and designing rules and constitutions used for generating training data for SFT and training preference models. A big open question is whether LMs-as-evaluators inherit human biases, even when carefully prompted. A simple experiment addressing this question would involve having an LM play the role of an evaluator and conducting a similar analysis for its judgements.

Author response:
It's true that "human evaluation with *crowdworkers*" is not always necesary, but language is an inherently human construct and so we would argue that humans are likely to be required to some degree for the foreseeable future. Using models for evaluation is more scalable but these models were still trained on data (ie language) that was originally generated by humans. Further, more targeted experiments would be valuable to confirm whether using LLMs as evaluators mitigates or exacerbates the biases we identify in this paper.

Reviewer comment (Evaluation):
The crowdworkers were asked to rate LM responses according to each criterion in series and then to give an overall score. I wonder if such ordering of the task is not itself a confounder. Would overall scores be different if the crowdworkers were not asked to give scores for criteria before? Would a score for `Formatting` be different if the crowdworkers were not asked for `Factuality` before? Do authors control for this anyhow?

Author response:
Yes, we accounted for this - see the first paragraph in Sec 2.1. We used one groups of annotators to collect overall preference scores, and a separate group for the error annotations (who were also asked for overall ratings). You are correct that there is a 'priming' effect: Fig. 7 in the appendix shows a higher correlation between error scores and overall scores for the group that performed the error annotation, indicating that this did indeed affect their decision. The main results in the paper are based on the 'unprimed' annotations from the separate group, but we consider this an auxiliary finding of interest.

Reviewer comment (Experiments):
This is not really a flaw, but it would be much better if the experiments were done on responses from frontier models (such as GPT-4, Claude 2 and possibly LLaMA-2-70B). Was budget the limiting factor?

Author response:
Our experiments were run using the most performant models available with permissive licenses at the time the experiments were run. GPT-4 and Claude were not available to us for legal reasons.

Reviewer comment (Experiments):
The paper feels a bit ad-hoc; the properties tested seemed kind of arbitrarily chosen, but the specific set tested should have a significant impact on the learned lasso weights (e.g., if one feature is more predictive than the rest, the lasso weight would put all of it on that)

Author response:
We disagree that Lasso will choose a single feature, since the features are not all perfectly correlated. Lasso will include any features that lead to a large enough predictive improvement to counteract the penalty term. However, we have updated the paper with more details of how the error types were chosen.

Reviewer comment (Experiments):
The RLHF increases assertiveness claim is tested indirectly; it compares Llama 2 13B and Command 52B, which have many confounders beyond RLHF

Author response:
Our results do show that Llama2 exhibits higher assertiveness at equivalent quality, but it's true that we cannot be certain *why* this is the case. Since the major difference between the two models is the RLHF training, we suggest that this may be the cause of the difference. 

One of our primary contributions is to clearly demonstrate the biasing effect of assertiveness on perceived quality. To the best of our knowledge, this is the first work that demonstrates this for LLMs and we hope this research will provide a basis for future efforts to build on. 

Thanks for your response and for increasing your score!

We did update the paper to be more explicit regarding the limitations of the claims regarding Llama and Command, it was not our intention to claim that this was due to the RLHF. As you rightly point out, we are trying to convey that RLHF may be one of the contributors affecting this observation, but there could of course be others that are not possible to isolate and test for. Thank you for your feedback and for helping to improve the quality and messaging of our work!

Reviewer comment (Experiments):
How much does the choice of ten properties you label for matter? What would happen if you included different properties?

Author response:
I'm not sure I fully understand the question - clearly, a different set of properties would lead to different results. However, our intention was to establish a set of error types with as much coverage as possible. We welcome any suggestions for errors that have been ommitted.

Reviewer comment (Novelty):
I wish there were discussion on not just the problem of individual judgments being ambiguous, but that standard RLHF is optimizing towards a single user preference per example rather than considering factors resulting in a distribution over judgments for a number of different annotators (a point that is made e.g., as a motivation for jury learning, Gordon et al. 2022).

Author response:
Thanks for the pointer to the jury learning paper, we have added this to our discussion. We briefly mention some related work on modelling annotator distributions by [Wang et al., 2023](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00584/117217). Collecting multiple annotations will partially mitigate the problem, but care still needs to be taken when aggregating to account for the biases that we present here. There are many [issues with RLHF](https://arxiv.org/abs/2307.15217) but our intention was to perform a targeted analysis of whether preference scores are a good training/evaluation metric.

Reviewer comment (Theory):
I would also like more discussion on what to do in light of these findings. There were hints at one possible perspective of a solution -- optimizing nebulous "preference" is far from what people do when using language, which is more based on utility and communicative success (which preference is not tied to at all).

Author response:
Thank you, this is a very valid point! Testing possible mitigation strategies is outside the scope of this paper, but we have added more detailed suggestions for possible future directions.

Reviewer comment (Experiments):
Following the above, I would have liked to see an experiment that directly shows how optimizing for human preference influences outputs across different dimensions explored in this paper (e.g., assertivness). The evidence from this paper is post-hoc analysis of existing models (in Section 4), and the earlier experiments showing annotators generally conflate quality with features like assertiveness/complexity. There are a couple of things this analysis conflates itself, e.g., the actual data that Command and Llama 2 are trained on, and their learning algorithms. But an even stronger experiment would perform RLHF-based finetuning (or some kind of finetuning with human preference judgments) on data where we have judgments of or could control complexity and assertiveness, and measure how the final models perform on factuality judgments after this process. E.g., perhaps this would show that optimizing for assertiveness++ responses hurts factuality. Or maybe it would show that with the exact same data, fine-tuning using a reward model exacerbates this problem more than just fine-tuning directly. But with the current analysis presented, it's hard to make any concrete conclusion beyond that all these models result in judgments conflating quality with style.

Author response:
We are the first to demonstrate the existence of a biasing effect due to assertiveness, which is not a trivial finding. We believe it is unfair to critique the paper for not also including an additional large-scale experiment investigating the effects of RLHF.

We disagree that the only conclusion is that "all the models conflate quality and style": the results definitely show that Llama 2 exhibits higher assertiveness at equivalent quality. However, it is a valid criticism that we cannot be sure *why* this is. We tried to qualify the findings in Section 4 as "preliminary evidence" since we did not directly test the same models with and without RLHF. A targeted set of experiments, using the same base model and varying the number of RLHF steps (and other factors), is planned for the future but exceeds the scope of this paper.

Reviewer comment (Evaluation):
I would also like to see an experiment controlling for factuality, like in Section 3.2. E.g., if it's possible to even post-hoc separate the results in Figs 3 and 4 by whether an expert judged the output as factual or not. Basically, do you find that assertive false responses are rated as factual by annotators more often than inassertive true responses?

Author response:
Here are the annotator factuality error rates, grouped by preamble type and expert factuality judgement:

| Preamble Type   |   Factual  |   Not Factual |
|:----------------|----------------------------:|----------------------------:|
| Assertiveness\-\-  |                     1.88679 |                     0       |
| Normal          |                     1.04 |                    16.66  |
| Assertiveness++ |                     2.17 |                     3.57 |


So, it's not the case that "assertive false responses are rated as factual by annotators more often than inassertive true responses", but the factual errors are much less likely to be spotted in assertive outputs. Note that there are only 7 examples that are inassertive and false, so the statistical power of this analysis is weak, and we therefore will not include it in the paper. Expert annotations are unfortunately difficult to scale!

Reviewer comment (Presentation):
I think the main table showing expert judgments for Sec 3.2 is in the appendix, but in my opinion it should be in the main paper.

Author response:
We would like to include the full results in the main flow of the paper, but unfortunately space is limited. We have the choice to either show annotator and expert judgements alongside each other and require the reader to take the difference, or to show just the difference and include the breakdown of results in the appendix. Presenting the results as differences demonstrates most clearly the findings we are trying to present, without excess complication, but we include the full detailed results in the appendix for completeness.

Reviewer comment (Experiments):
Why were the error types in Section 2 the ones chosen? There is a brief discussion (inspiration from Xu et al. 2023, Gricean maxims) but it would be nice to have more discussion on why these particular types were chosen.

Author response:
Thanks for the useful feedback! We have updated the paper to include more detail on how the error types were selected.

Reviewer comment (Presentation):
Why do contradiction/scope/fluency/harmful have no contribution in Figure 1?

Author response:
We address this at the top of Page 4 - scope/fluency/harmful errors were very rare, while contradiction seemingly doesn't have a strong enough influence on overall rating to be selected by the Lasso model.

Reviewer comment (Evaluation):
Did you analyze where disagreements came from in factuality judgments in Section 2.1? Where agreement was 0.64

Author response:
We did not attempt to categorise the types of disagreement. However, based on our experience from performing the expert annotations ourselves, some statements in outputs required significant effort to verify (e.g., checking a software user manual to confirm whether a menu item exists or not), and other statements were plausible but wrong.

Reviewer comment (Experiments):
What was the hypothesis of assertiveness and complexity being confounders founded upon? Are there other confounding factors that could influence this, e.g., length?

Author response:
Yes, other confounding factors are likely to exist! Length does have an effect on scores, we include an analysis in Fig 8 in the appendix. As we mention in the text, this is complicated by the fact that long answers are more repetitive (and therefore more annoying) and short answers simply contain fewer facts that could be incorrect. We selected assertiveness and complexity as they are intuitive yet undesirable properties of LLM outputs, but there could be other factors worth investigating in future.

Reviewer comment (Evaluation):
"incorrectly perceived as being less repetitive" -- is repetitiveness objective? It seems a lot more subjective than something like factuality.

Author response:
You are correct, repetition is not an entirely objective property. However, the instructions refer to repetition of phrases or items in a list, so it is more tightly defined in the context of our experiments.

Reviewer comment (Presentation):
What are the two lines in Figure 6?

Author response:
Thanks for the feedback - these are the trendlines for Command 52 and Llama 2. We have updated the caption accordingly.

Reviewer comment (Experiments):
Figure 5 shows annotated error rates -- are these expert annotations or general crowdsourced annotations?

Author response:
These are crowdsourced annotations - we have updated the caption accordingly.

Reviewer comment (Experiments):
it is not clear if this error type categorization is comprehensive enough; authors haven't provided any empirical/experimental support for these error categories.

Author response:
Thank you for the feedback! The error categories were based partly on the well-established Gricean Maxims (e.g. the Maxim of Quantity prohibits repetition), partly on experimental work by [Xu et al.](https://aclanthology.org/2023.acl-long.181/) who asked human evaluators to give justifications for their scores, and influenced by the factors that users care about when using LLMs in production environments. We did our best to account for as wide a range of errors as we could, but it's possible that we have missed something. We have updated the paper with more details.

Reviewer comment (Experiments):
There are other important biases from AI safety perspective that isn't explicitly studied in the paper (could possibly make the paper more relevant) - hallucinations, more fine-grained categorization of inconsistencies, toxicity, etc.

Author response:
Safety is an important area of research, and we agree that there would be a lot of value in extending our study to consider the interaction between assertiveness/complexity and various types of unsafe LLM behaviour. However, we believe that doing this justice would be another whole paper in itself.

In this paper we confirm that preference scores are a) an imperfect measure of important error types, and b) biased by assertiveness. These observations hold true even for a (hypothetical) perfectly safe model. We hope it will serve as a starting point for further work, including experiments specifically targeted at the wide range of important safety concerns.

Reviewer comment (Experiments):
the datasets studied may not be comprehensive enough to have captured some of the important dimensions of AI safety (toxicity, attribution, etc).

Author response:
We selected datasets that are well-scoped (such that evaluation is possible) but are still challenging and require long-form generation. Further work is definitely needed - but there is a lack of datasets with permissive licenses for realistic use cases.

Reviewer comment (Experiments):
authors note that human preferences are mostly absolute or relative - there isn't any comparison between the two in this research. Since the paper provides insights into understanding these human preferences, more focus on various types of human preferences would have strengthened the finding of the paper.

Author response:
Time and budget constraints meant that we used a fixed protocol, based on RankME (Novikova et al., 2018), and focussed on identifying whether the biasing effects exist at all. You're right that it would be really interesting to run our experiments with different protocols: varying the number of outputs evaluated at a time, using absolute/relative scores, different instructions etc. Future work could investigate whether the strength of the effect varies with annotation protocol.

Reviewer comment (Evaluation):
Authors very briefly touch upon the AI safety implications but do not give justice to this very important piece of information in the paper - what are the AI safety implications as a result of using human preferences for training and evaluation.

Author response:
Safety is an important area of research, and we agree that there would be a lot of value in extending our study to consider the interaction between assertiveness/complexity and various types of unsafe LLM behaviour. However, we believe that doing this justice would be another whole paper in itself.

In this paper we confirm that preference scores are a) an imperfect measure of important error types, and b) biased by assertiveness. These observations hold true even for a (hypothetical) perfectly safe model. We hope it will serve as a starting point for further work, including experiments specifically targeted at the wide range of important safety concerns.