Paper: The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR-2024)

Reviewer comment (Experiments):
As I am more interested in the benchmark dataset itself, the evaluation for dataset validation seems limited. **The author should focus on providing a reliable benchmark as the major scientific contribution**; I believe the finetuning methods discussed in the manuscript are not as significant. Therefore, the authors should provide more evaluation results on more LLMs (open-source + closed-source) to validate your benchmark dataset. The evaluation results on popular models like GPT-4 would be very helpful. Although you can't get the weights, test by prompting would be sufficient. I am also curious to know the difference in consistency between the pre-trained model vs their SFT-ed version (i.e. llama2-7b vs llama2-7b-chat). If the compute resource allows, the parameter scaling on reward consistency could also be an interesting point for investigation.

Author response:
For GPT4’s results on our Contrast Instruction, please refer to our answer to your **Q1**.

RM consistency does not equal the consistency of the final DPO-trained LLM, that’s why we use the ‘trickle-down effect’ in our title. However, exploration on SFT/RLHF LLMs would be interesting, we have explored potential methods for assessing preference consistency in LLMs trained via SFT/RLHF, each with its own limitations:
- Prompting DPO-trained LLMs to choose the better response often leads to them not following the instruction correctly, such as repeating the instruction instead.
- Calculating the likelihood of instruction-response pairs is hindered by length bias, with DPO-trained LLMs tending to favor longer responses, resulting in poor performance on preference benchmarks.

Reviewer comment (Reproducibility):
The benchmark dataset should be submitted along with the paper, as I believe this is the core contribution of the paper.

Author response:
We’ve just uploaded the data in the system, feel free to check.

Reviewer comment (Evaluation):
How concerned are you about the risk of data leakage? What implications would arise if instances from the benchmark dataset were also present in LlaMa-7B's pre-training data or the dataset used to train the reward model? Have such overlaps impacted the assessment of reward consistency?

Author response:
It’s a good question. In our study, specific measures to address data leakage were not the primary focus. However, we understand the importance of this issue and will ensure rigorous checks against data leakage based on the very recent work [1,2]. We agree that understanding the impact of data leakage on reward consistency is crucial, as highlighted by our use of CONTRAST INSTRUCTIONS to enhance RM consistency​​. 

But we couldn’t complete the analyses towards whether LLaMa pre-training corpus contains our data during the rebuttal period.

[1] Detecting Pretraining Data from Large Language Models
[2] Proving Test Set Contamination in Black Box Language Models

Reviewer comment (Experiments):
I need more details on Section 4. For "Single-Task", what is the split between train and test?

Author response:
Certainly. The term 'single-task' in our study refers to training our RM exclusively on data from one particular task-specific benchmark and subsequently testing it against the Contrast Set derived from the same task. For each task, we partition the data into training, development (dev), and test sets using an 8:1:1 ratio. We then identify and utilize the checkpoint that delivers the best performance on the development set for our final evaluation.

Reviewer comment (Experiments):
For Section 4, why would you fine-tune a benchmark dataset? And the fact that it performs so poorly after being trained and tested on the same dataset distribution is surprising. The consistency improvement seems marginal.

Author response:
In Section 4, we detail the process of fine-tuning our RM using the training sets from task-specific benchmarks and then evaluating it on our Contrast Set. Our results, presented under 'single-task' and 'multi-task' training conditions, clearly indicate that multi-task training does not enhance RM consistency. 

Moreover, Table 2 shows the RM's performance on the Contrast Set, while Table 8 provides insights into its performance on the original test sets, where the RM demonstrates an average accuracy exceeding 75%. This is a huge gap that reflects how severe the RM inconsistency issue is.

Besides, we believe that our Contrast Set and the original test sets should not be entirely viewed as the same distribution. 
- The original benchmarks feature responses that are relatively easy to differentiate due to lower lexical similarity. 
- Our Contrast Set comprises pairs that are lexically similar but semantically diverse, posing a significant challenge to the RM's capacity for nuanced semantic comprehension. 

This distinction is crucial as it demonstrates that achieving high accuracy on the original test sets does not necessarily equate to high performance on the more demanding Contrast Set.

Reviewer comment (Theory):
The paper claims the issue in reward inconsistency is due to the reward over-optimization issue (citing Gao et, al). Still, the reason for such reward over-optimization and how that causes the inconsistencies in the particular Contrast Dataset is unclear. More specifically the author claims that "From the RMs’ perspective, correctly distinguishing between a clearly good vs. bad response is easy". But the contrast dataset for example shown in Figure 1, they are very different questions although textually there are common words. Does that mean the current LLMs are not able to produce representations that can separate the two is not very clear and needs further clarification. For ex: "A is a good student" and "A is a bad student", they have a lot of words in common but in representation space they should be extremely different and should be trivially separated if the representations are reasonable. Thus it's not very clear how Contrast Dataset is providing a challenging dataset to test RM model inconsistency.

Author response:
**A1**: 

Please note that we are **NOT** claiming the specific benchmark itself is challenging for all reward models or LLMs. What we are claiming here is the **Contrast Instructions** benchmarking strategy, when applied to a specific RM training set to create an evaluation set, can reveal inconsistencies of the resulting RMs. 

First, we want to clarify that the (short) example shown in Figure 1 is mostly for illustration purposes due to space constraints. Most examples in contrast instructions involve much longer responses that are difficult for the models to distinguish – some of the examples can be found in Table 1. 

But even with the “A is a good student” vs. “A is a bad student” example, the representation would be “extremely different” **only when your model is properly trained to generalize to the task that it’s solving**. e.g. sentiment classification. Let’s say if the task prompt asks “how topically relevant are the two sentences”, the two would have similar representations. Here you are making the assumption that the model generalizes well from training, and so it would be sensitive to semantic changes even when there’s a lot of lexical overlap between responses. What we intend to show with our $C_{res}$ and $C_{ins}$ metric is exactly that RMs do NOT generalize well with the standard training objective. In NLP, similar generalization issues and insensitivity to examples with high lexical overlap has been observed across different settings [e.g. 1, 2]. We are observing similar patterns with RMs as well.

Reward Over-optimization – For example, within the open-source community (e.g. w/ Llama1 and 2), examples from StackExchange make up for majority of the RM training data.  StackExchange follows a specific “proxy” way of defining human preferences, e.g. within the same question, each human preferences example are constructed from responses to same question/thread, and responses with the highest votes are considered the preferred responses. What we intend to show with our $C_{res}$ metric is that RMs trained this way only fits the training distribution, and don’t generalize to what we think RM should be doing – i.e. being able to distinguish between good/bad responses consistently. With contrast instructions, we observe that RMs fail to do so even when the questions come from the domain (e.g. StackExchange) as seen in training. 

[1] Gardner, Matt, et al. "Evaluating Models’ Local Decision Boundaries via Contrast Sets." Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

[2] McCoy, Tom, Ellie Pavlick, and Tal Linzen. "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

Reviewer comment (Experiments):
Another point is that why such reward inconsistencies are not observed in standard available RLHF datasets like Carper AI, hh, etc. are not made explicit. Does it mean that the majority of the datasets are easily separable and lacks samples towards the optima which is hard for human to segregate? A comparison with current RLHF methods is critical on standard datasets to understand the significance and mitigation of the problem properly.

Author response:
**A2**: 
Thanks for your question. We're claiming the specific benchmark itself is challenging for reward models. What we are claiming here is the **Contrast Instructions** benchmarking strategy. 

Moreover, we don’t claim open benchmarks are flawed, we don’t observe consistency issues on them due to the following differences. 
In our contrast set, we use golden response from a similar instruction as a negative example, which can be regarded as out-of-distribution samples and make RM difficult to distinguish.
In standard benchmarking, their testset are more in-domain datasets compared to training dataset, which makes RM easier to recognize.

Regarding open benchmarks and models, as specified in our experimental settings, we use the official RM checkpoints (Stack-LLaMa) from Huggingface RLHF team (https://huggingface.co/trl-lib/llama-7b-se-rm-peft), which is trained on open-source benchmark Stack-Exchange. Our results show that it still suffers from severe inconsistency issues.

Reviewer comment (Theory):
"Surprisingly, we observe close to random-chance performance ..while humans are able to rank the responses correctly in ≈ 80% of the cases" So, is it the case that humans are able to identify it but the reward models are not able to learn the same and failing is not very clear since its a supervised learning problem and can be shown to be strongly convex under certain settings. Hence, a clear description is missing on the same, and will be interesting to have a discussion with reference to the recent works showing convergence [1,2] and where this issue can arise in that context.

Author response:
**A3**: From a generalization point of view, what our metric $C_{res}$ is measuring is not what the model is being trained on exactly. See our response above for “Reward Over-optimization”. It’s a supervised learning problem, but again, what we are claiming here is the poor generalization abilities with RMs trained under this protocol.

As for the derivations from the theoretical works in [3,4] or convexity in your comment, we believe they are quite distant from real-world scenarios:
- Non-convexity. In the context of RM, we aim to optimize towards human preference. Optimizing towards human preference is more than being a convex optimization problem, which has shown in tons of works in RLHF literature.
- Misaligned targets: [4] aims to propose a framework that tackles challenges in policy alignment in reinforcement learning, instead of reward modeling, which is the main focus in our paper. Despite its contributions, theoretical analysis in [4] is not very helpful to give insights about the reward model we discuss in this paper. 
- Misaligned targets: [3] shows that when the true reward function is linear (in their abstract and intro), the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However, we know that the reward model trained from human preference is non-linear.

- Strong assumptions. [3] assumed that the policy trained is greedy with respect to the learned reward. However, in practice the reward is mostly used to fine-tune the pre-trained policy, as claimed in their conclusions.

Reviewer comment (Evaluation):
ContrastDataset provides is challenging dataset, how are humans able to do good on it? Is it mainly hard for the LLMs since the representations are sub-optimal?

Author response:
**A5**: While Reward Models (RMs) currently struggle to differentiate patterns within Contrast Instructions, this challenge is not mirrored in human performance. Humans are adept at discerning subtle semantic differences, an ability that RMs have yet to develop fully. 

This disparity highlights the difficulty RMs face in learning representations that effectively separate human-preferred responses from non-preferred ones. At least, our results show the limitation of the main-stream protocol for modeling RM, thus motivating more sophisticated designs for RM training.

Reviewer comment (Experiments):
Recent work on Direct Pref Optimization (DPO) shows that it can learn with learning reward models, will such an issue happen there as well?

Author response:
**A6**: Given that DPO algorithms do not incorporate an explicit RM, evaluating their consistency in the same manner as RMs is not feasible.


Moreover, RM consistency does not equal the consistency of the final DPO-trained LLM. That’s why we use ‘trickle-down effect’ in our title. However, we have explored potential methods for assessing preference consistency in LLMs trained via DPO, each with its limitations:
- Prompting DPO-trained LLMs to choose the better response often leads to them not following the instruction correctly, such as repeating the instruction instead.
- Calculating the likelihood of instruction-response pairs is hindered by length bias, with DPO-trained LLMs tending to favor longer responses, resulting in unfair evaluation on preference benchmarks.

Reviewer comment (Evaluation):
I was not sure of whether the contrast instructions generated automatically do mean something different where one answer is strictly better than the other. The authors say they restricted this based on a particular cosine distance range, but some more rigorous evaluation around this could have made the impact of the dataset a bit stronger. For instance, one can query a bigger open LLMs in order to validate that the contrast instructions are fit for purpose. The other option could have been to conduct human evaluation on a randomly selected subset of the contrast instructions dataset.

Author response:
**A1**: Thank you for the comments. We have human evaluation results on $C_{res}$ on each dataset in table 8 of Appendix B. Due to space constraints, we only reported human performance averaged over 4 datasets in Table 2. 
 
After reading the reviews, we conducted a more thorough human evaluation on each of the dataset. On average over 4 datasets , we see that humans rank responses correctly ~82% of the time. If we allow for Google search to access relevant knowledge, the accuracy goes up to 96%. We also conducted the same evaluation with GPT4 prompting with 4-shot demonstrations, from which we observed 95% accuracy in terms of $C_{res}$. We think the gap between RM vs. human performance helps justify our observations on RM inconsistency. 

Regarding cosine range, we have consideration for the following factors:
- Upper Bound: We set the upper limit (e.g., 0.9) to avoid retrieving identical instructions. This is crucial to ensure  that $I_{A} \circ r_{A}$ is better than $I_{B} \circ r_{A}$.

- Lower Bound: We avoided setting the lower bound too low (e.g., 0.5) to maintain the challenge for the RM. A lower range would make it too easy for the RM to judge, failing to test whether RM has a nuanced understanding of human preferences.

- Range Interval: We didn’t opt for a narrow range (e.g., [0.85, 0.9]) as it would greatly limit the size of our Contrast Set.

To satisfy all of three factors, we can only handcraft a proper range, so we set an upper limit (e.g. 0.9) to avoid retrieving identical instructions, and the lower limit (e.g. 0.75) is adjusted mostly to make sure we can sample enough test examples in each human preference dataset.   

We have updated the human eval and GPT4 results in Appendix G, where both human and GPT4 obtain over 90% accuracy on our Contrast Instruction. We believe these results can serve as a quality check for the data built by our Contrast Instruction benchmarking strategy. Moreover, we’ll try to allot more space to show human performance in the main body as well.

Reviewer comment (Experiments):
The authors could have just chosen one single baseline LLAMA 7B and used that to validate their claims. The behaviour of LLAMA 7B could be grounded in the kinds of datasets and training procedures that it was subject to. Having a few other RM baselines trained with similar RM training loss could have made the claims even stronger.

Author response:
**A2**: Thanks for you this suggestion! – In Appendix H, we've added experiments with RMs trained with backbone models of different sizes. We observe that scaling model size leads to (1) increased test performance on the original human preference dataset, but (2) the $C_{res}$ and $C_{ins}$ performance remains flat. This echoes with our hypothesis that – standard RM training fits the only the training distribution, but do not lead to a generalized model that can distinguish good/bad responses consistently. Just like what our general response said, this RM consistency issue is model-agnostic.

Reviewer comment (Evaluation):
The human evaluation was done on 100 randomly selected data points and no standard errors and variance numbers were reported around the results. Since some of the results are close to each other, having the error bands around the results should help clarify the significance of improvements.

Author response:
**A3**: Thank you for your comments. 

Regarding your suggestion, we recruit two emergency Computer Science expert annotators (as understanding StackExchange responses requires domain knowledge in CS). The new results are shown in Appendix J, which aligns with our human evaluation. Besides, we provide the Cohen kappa correlation to show consistency between human annotation, which show that these two human evaluations share a certain agreement.

Reviewer comment (Evaluation):
One other concern about human evaluation is the possibility of potential bias as the authors themselves serve as human annotators for human evaluation of the results. There are several important statements made in this paper based on human evaluation. I would have preferred at least a mixture of external annotators (who are not aware of the work or how the responses were generated) to de-bias the evaluations. "Finally, we report human performance resulting from the majority vote of three human annotators (the authors) on 100 randomly selected data points."

Author response:
**A3**: Thank you for your comments. 

Regarding your suggestion, we recruit two emergency Computer Science expert annotators (as understanding StackExchange responses requires domain knowledge in CS). The new results are shown in Appendix J, which aligns with our human evaluation. Besides, we provide the Cohen kappa correlation to show consistency between human annotation, which show that these two human evaluations share a certain agreement.

Reviewer comment (Experiments):
There were certain choices of methods made in terms of methods for response/instruction similarity and the dataset construction, including the augmented data points for ConvexDA. Have other methods been evaluated and ablated against before choosing these methods? I wanted to understand if we are doing the best we can in terms of construction of these datasets and methods.

Author response:
**A4**: In our definition of inconsistency, only when reward scores shift to the opposite direction of quality changes can be considered inconsistent issues.
When constructing a benchmark for evaluating RM consistency, there are mainly two challenges: 
- (1) guarantee: we need to guarantee that $I_{A} \circ r_{A}$ has higher qualities than $I_{A} \circ r_{B}$. 
- (2) automation: we need to find an affordable and scalable benchmarking strategy.

We have tried other alternatives. 

(a) Doing data augmentation on responses/instructions (i.e., as suggested by you), but it is difficult to guarantee how the quality changes after data augmentation, which breaks (1). 

(b) Recruit human experts to annotate the Contrast Instructions. This would be ideal, but it remains too expensive considering our limited budget, which breaks (2).

Reviewer comment (Experiments):
I know you have considered the classic RM loss, log (sigmoid (R_chosen - R_rejected)). Have you considered other loss functions such as the margin loss used in LLAMA 2? I am not specifically looking for one kind of loss, but just a choice of a few other RM losses. Considering different losses can help us understand whether RM inconsistency arises from the dataset or the kind of losses used or a combination of both. This would be a good ablation to understand where RM developers should focus more of their efforts on.

Author response:
**A5**: Thanks for your suggestion! We retrain RM using margin loss (adding margin $r_{m}$ just like LLaMa2 paper), where the margin is selected based on the dev set. The results are shown as follows:

|  | Stack | WMT | Twitter | Sum |
|:---:|:---:|:---:|:---:|:---:|
| no margin | 50.1 | 56.4 | 60.1 | 47.7 |
| with margin | 51.1 | 56.8 | 59.8 | 50.1 |

The results show that the margin loss provides very limited improvements to the baseline. Overall, we believe the challenge of reward consistency is relevant to various factors, including model architecture, loss function, data, etc.

Reviewer comment (Experiments):
A lot of times, RM training can easily lead to overfitting to the training datasets and creating a generalization gap in the process. Can you kindly explain how you do model selection for RM training for the baseline and your variants?

Author response:
**A6**: In our experiments, we prepare a devset for each benchmark. We select the checkpoint that achieves the optimal result on the dev set. Specifically, the dev set is randomly sampled and split from the original test set. Moreover, we also have a devset for our Contrast set, based on our observations, the checkpoints generally perform badly on the devsets of the Contrast Set.

Reviewer comment (Evaluation):
In Table 2, I was not sure what (estimated) human performance means. Can you kindly clarify?

Author response:
**A8**: The estimated human performance was derived from a majority vote among the annotators on a subset of data points, due to our limited budget. Therefore, considering being rigorous, we say estimated human performance here.

Reviewer comment (Experiments):
It's a little unclear why CONVEXDA is used as a way to robustify the RM. It seems that it might work because it specifically targets the proposed benchmark, which mainly tests inconsistent instruction-response pairs with lexically similar words, but I'm not sure if the method can work well in scenarios where inconsistent responses/instructions are beyond just lexically similar words with different meanings. Moreover, I wonder if it would work equally well to simply use a different LLM to generate those similar pairs for augmentation.

Author response:
**A1+A2**: One can embed lexical variations in training when training RMs, but that is not the point. Our goal was to introduce methodologies to improve RM consistency in a manner that is **agnostic** to the choice of evaluation (i.e., not presupposing any knowledge of the patterns inherent in Contrast Instructions), thereby ensuring a fair evaluation.

Our proposal approaches (ConvexDA and RewardFusion) are not informed of the pattern Contrast Instructions, and serve as general augmentation methods. Moreover, they both contribute to performance enhancements on the original test sets, as evidenced in Table 7 (RMEval) and Table 8 (Original test set), indicating that they are beneficial beyond the Contrast set pattern. Therefore, our approaches are designed not to be informed by the patterns specific to Contrast Instructions. They are crafted to address the general generalization of RM, like other data augmentation tricks. This overarching goal explains why the improvements (on the Contrast set) attributed to these methods are incremental.

Reviewer comment (Novelty):
I'm also less clear about why REWARDFUSION is needed and how it contributes to fixing inconsistency. Again, it seems to mainly target the type of inconsistency due to lexically similar words with different meanings, which may not be general enough.

Author response:
**A1+A2**: One can embed lexical variations in training when training RMs, but that is not the point. Our goal was to introduce methodologies to improve RM consistency in a manner that is **agnostic** to the choice of evaluation (i.e., not presupposing any knowledge of the patterns inherent in Contrast Instructions), thereby ensuring a fair evaluation.

Our proposal approaches (ConvexDA and RewardFusion) are not informed of the pattern Contrast Instructions, and serve as general augmentation methods. Moreover, they both contribute to performance enhancements on the original test sets, as evidenced in Table 7 (RMEval) and Table 8 (Original test set), indicating that they are beneficial beyond the Contrast set pattern. Therefore, our approaches are designed not to be informed by the patterns specific to Contrast Instructions. They are crafted to address the general generalization of RM, like other data augmentation tricks. This overarching goal explains why the improvements (on the Contrast set) attributed to these methods are incremental.

Reviewer comment (Experiments):
Please clarify how general the CONVEXDA and REWARDFUSION are in fixing inconsistency beyond lexically similar but different words.

Author response:
**A1+A2**: One can embed lexical variations in training when training RMs, but that is not the point. Our goal was to introduce methodologies to improve RM consistency in a manner that is **agnostic** to the choice of evaluation (i.e., not presupposing any knowledge of the patterns inherent in Contrast Instructions), thereby ensuring a fair evaluation.

Our proposal approaches (ConvexDA and RewardFusion) are not informed of the pattern Contrast Instructions, and serve as general augmentation methods. Moreover, they both contribute to performance enhancements on the original test sets, as evidenced in Table 7 (RMEval) and Table 8 (Original test set), indicating that they are beneficial beyond the Contrast set pattern. Therefore, our approaches are designed not to be informed by the patterns specific to Contrast Instructions. They are crafted to address the general generalization of RM, like other data augmentation tricks. This overarching goal explains why the improvements (on the Contrast set) attributed to these methods are incremental.

Reviewer comment (Experiments):
Please compare to methods that simply use LLMs for data augmentation.

Author response:
**A4**: In Appendix F, we present a comparison between ConvexDA and standard data augmentation (DA) techniques, such as paraphrasing. ConvexDA is not aimed at outperforming traditional data augmentation methods in terms of raw performance. Figure 8 shows that ConvexDA achieves the desired augmentation effect with greater efficiency, making it a superior choice when considering the trade-offs inherent in the data augmentation process.

Reviewer comment (Experiments):
Contrast instructions benchmark: To sample lexically similar but semantically different pairs of instructions the authors only sample instruction pairs that lie within the similar range of 0.75 and 0.9. It would be useful to explain the sensitivity of this hyperparameter and study how well this prevents sampling semantically similar instructions.

Author response:
**A1**: We hand-crafted our cosine similarity range for the following reasons:
- Upper Bound: We set the upper limit (e.g., 0.9) to avoid retrieving identical instructions. This is crucial to ensure  that $I_{A} \circ r_{A}$ is better than $I_{B} \circ r_{A}$.

- Lower Bound: We avoided setting the lower bound too low (e.g., 0.5) to maintain the challenge for the RM. A lower range would make it too easy for the RM to judge, failing to test whether RM has a nuanced understanding of human preferences.

- Range Interval: We didn’t opt for a narrow range (e.g., [0.85, 0.9]) as it would greatly limit the size of our Contrast Set.
To satisfy all of three factors, we can only handcraft a proper range.

In Appendix G, we include a human analysis on each of the dataset, to see how well humans can do in terms of $C_{res}$ and $C_{ins}$. On average over 4 datasets , we see that humans rank responses correctly ~82% of the time. If we allow for Google search to access relevant knowledge, the accuracy goes up to 96%. We think the gap between RM vs. human performance helps justify our observations on RM inconsistency, as well as contrast instructions as a benchmarking strategy.

Reviewer comment (Experiments):
When evaluating existing RM on this benchmark the reward models get a low C_res score of 53.6, even though C_res conceptually resembles the RM learning objective. This is surprising given the benchmark was constructed using the same datasets used for training. It would be useful to verify if appropriate hyper-parameter tuning was performed and if the model is able to overfit and get a high C_res score.

Author response:
**A2**: We want to clarify that – $C_{res}$ only **conceptually** resembles the RM training objective, as in, both involve distinguishing between two responses given an instruction. In practice, however, RM training data is usually constructed using “proxies” of human preference labels. For example, within the open-source community (e.g. w/ Llama), examples from StackExchange make up for the majority of the RM training data.  StackExchange follows a specific “proxy” way of defining human preferences, e.g. within the same question, each human preferences example is constructed from responses to the same question/thread, and responses with the highest votes are considered the preferred responses. What we intend to show with our $C_{res}$ metric is that RMs trained this way only fits the training distribution, and don’t generalize to what we think RM should be doing – i.e. being able to distinguish between good/bad responses consistently. With contrast instructions, we observe that RMs fail to do so even when the questions come from the domain (e.g. StackExchange) as seen in training.    

To your concern about our implementation – In our experiments, we actually tested on an **off-the-shelf** open-source RM trained on StackExchange data– Stack-LLaMa from the Hugging Face RLHF team (https://huggingface.co/trl-lib/llama-7b-se-rm-peft). The results are shown in Table 3. We see close to random performance from the off-the-shelf RM under $C_{res}$ as well.

Reviewer comment (Experiments):
Limited models and scales: It would be useful to explore more pretrained models and perform a model and dataset scaling analysis to measure if and how the consistency of reward models varies across different models and scales and if similar observations are found. Would be useful to do a similar analysis when evaluating the impact of the proposed improvement methods ConvexDA and RewardFusion to see how well they perform with different models.

Author response:
**A3**: Thanks for your suggestion – In Appendix H, we've added experiments with models of different sizes. We observe that scaling model size leads to (1) increased test performance on the original human preference dataset, but (2) the $C_{res}$ and $C_{ins}$ performance remains flat. This echoes our hypothesis that – standard RM training fits only the training distribution, but does not lead to a generalized model that can distinguish good/bad responses consistently.

Reviewer comment (Writing):
Please clarify what “finetuning with contrast instructions format” consists of.

Author response:
**A5**: The Contrast Instruction format is identical to the original benchmark, where each pair has an "Instruction" and "response" joined together. We've corrected this writing to clear up any potential confusion.

If you have any other questions, please feel free to ask.

Reviewer comment (Reproducibility):
Please clarify if ConvexDA was used to train the reward models for this stage as mentioned in the appendix.

Author response:
**A4**: We apologize for a typo in Appendix C related to our experimental setup. In our research, we fine-tuned the consistent RM with our contrast set without ConvexDA. Using ConvexDA and RewardFusion to improve the RM for RLHF didn't significantly improve the chatbot's performance due to their limited improvements in enhancing RM consistency. Since we focus on the trickle-down effect of RM consistency, we select the most effective method (training on Contrast Instruction) in RLHF to better observe such effects.

Reviewer comment (Evaluation):
Could you explain the reason behind the very low C_res score of the trained reward model on the same dataset, even though C_res resembles the RM training objective?

Author response:
**A2**: We want to clarify that – $C_{res}$ only **conceptually** resembles the RM training objective, as in, both involve distinguishing between two responses given an instruction. In practice, however, RM training data is usually constructed using “proxies” of human preference labels. For example, within the open-source community (e.g. w/ Llama), examples from StackExchange make up for the majority of the RM training data.  StackExchange follows a specific “proxy” way of defining human preferences, e.g. within the same question, each human preferences example is constructed from responses to the same question/thread, and responses with the highest votes are considered the preferred responses. What we intend to show with our $C_{res}$ metric is that RMs trained this way only fits the training distribution, and don’t generalize to what we think RM should be doing – i.e. being able to distinguish between good/bad responses consistently. With contrast instructions, we observe that RMs fail to do so even when the questions come from the domain (e.g. StackExchange) as seen in training.    

To your concern about our implementation – In our experiments, we actually tested on an **off-the-shelf** open-source RM trained on StackExchange data– Stack-LLaMa from the Hugging Face RLHF team (https://huggingface.co/trl-lib/llama-7b-se-rm-peft). The results are shown in Table 3. We see close to random performance from the off-the-shelf RM under $C_{res}$ as well.

Reviewer comment (Reproducibility):
Could you explain what finetuning on contrast instruction format dataset involves?

Author response:
**A5**: The Contrast Instruction format is identical to the original benchmark, where each pair has an "Instruction" and "response" joined together. We've corrected this writing to clear up any potential confusion.

If you have any other questions, please feel free to ask.