Paper: Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Why were Dolly, User-orient, and SuperNI selected as the tasks of interest?

Author response:
**Q:** Why were Dolly, User-orient, and SuperNI selected as the tasks of interest?

**A:** We thank the reviewer for their insightful question on our dataset setup. As mentioned in Section 3, the instructions are designed to present a collection of queries that could potentially be posed to a text-based AI assistant in real-world scenarios. We clarify that Dolly, User-Orient, and SuperNI are all human-written instruction datasets. Specifically, Dolly and User-orient are open-ended instructions and responses while we select a subset of SuperNI datasets that cover a wide range of public NLP datasets with long responses (Table 11). We also point that there is no consensus within the community about the datasets to be used for feedback acquisition and subsequent reward modeling. Hence, we focused on high quality datasets that are used for instruction tuning.

Reviewer comment (Experiments):
The generalizability of the results may be limited by the choice of model in the various experiments: (1) only Alpaca-7b was tested in terms of generating candidate responses, (2) only GPT-3.5-Turbo was evaluated as an AI annotator, (3) only LORA Alpaca-7b was selected as the reward model, and (4) win-rate was computed only against DaVinci-003. It would be helpful for the authors to clarify why those models were selected in each part of the paper.

Author response:
**Q:** Additionally, the generalizability of the results may be limited by the choice of model in the various experiments.

**A:** We thank the reviewer for their query regarding our experimental setup. 

- While it is indeed possible to change various components of the setup, the critique holds for any LLM paper as LLMs are rapidly evolving and increasing in number. Our choices reflect (a) strong models and practices which have had numerous pick ups by the community at large (e.g., GPT-3.5-Turbo, Alpaca-7B), (b) finite compute and labeling budget available for conducting the experiments in an academic setting.
- In our experiments, we generated $30$K candidate responses and got them annotated by humans and AI which costs ~$2000 USD in total. Although it would be interesting to generate candidate responses from other models, the subsequent annotation is time-consuming and out of our academic budget. 
- We also clarify that the win-rate against davinci-003 has been a de facto evaluation metric for many influential works on LLM alignment [1,2,3,4] We will mention this argument in the revised paper.

Following up on the reviewer’s suggestions, we do run additional experiments for (a) using GPT-3.5-Turbo for consistency analysis and (b) using LORA Alpaca-7B as a reward model.

(a) **GPT-3.5-Turbo for Consistency Analysis**

By default, we use GPT-3.5-Turbo with temperature = 0. Here, we perform an inconsistency analysis on 500 comparisons between the ratings and rankings feedback for (i) GPT-3.5-Turbo at temperature = 0.5, (ii) GPT-3.5-Turbo-0613, (iii) GPT-4. Here are the results:

| Model                                          | Inconsistency |
|------------------------------------------------|---------------|
| ChatGPT-3.5-Turbo-Temperature=0 (Ours) | 58%           |
| ChatGPT-3.5-Turbo-Temperature=0.5              | 56%           |
| ChatGPT-3.5-Turbo-0613                         | 54%           |
| GPT-4                                          | 50%           |

The standard error for the above numbers is 4% with 95% confidence. We find that all the model choices suffer from the inconsistency problem. This indicates that our results are not restricted by the choice of the AI annotator, and are indeed observable in different models too. 

(b) **Choice of the reward model**

- We clarify that there is no clear consensus on the “right” choice of the reward model. We believe that any good language understanding model could act as a reward model. Prior works [1,6] use decoder-only architecture for the reward model. These models are normally at a scale of 10-100 billion parameters and harder to train in the compute efficient manner without engineering tricks. Hence, we chose Alpaca-7B in our setup, and LoRA is just a method to finetune it in a parameter efficient manner. 
- On the other hand, [5,7] have used BERT transformer encoder models, which usually have less parameters, say 500M-1B parameters. Finetuning BERT models achieve very good performance on many NLP tasks. Hence, we repeat train reward models on the Roberta-large architecture.
- Specifically, we finetune a robert-large model on the ratings data using the regression loss. In addition, we train another roberta-large model on the rankings data using the negative log likelihood loss. We use these two finetuned models to perform Best-of-64 policy. Subsequently, we use GPT-3.5-Turbo to provide rating and ranking feedback for the rating/ranking Best-of-64 policy.

We present the results below:

Reviewer comment (Evaluation):
How would results in Figure 3 differ across tasks?

Author response:
**Q:** How would results in Figure 3 differ across tasks?

**A:** We thank the reviewer for suggesting we perform the task-specific analysis of the evaluation inconsistency.  Here, we add the results:

| OASST        | Best-of-64 (Rating) | Best-of-64 (Ranking) |
|--------------|---------------------|----------------------|
| Ranking Eval | 46.7                | **52.6**                 |
| Rating Eval  | **47.0**                | 43.5                 |
|              |                     |                      |
| Vicuna       | Best-of-64 (Rating) | Best-of-64 (Ranking) |
| Ranking Eval | 48.6                | **54.3**                 |
| Rating Eval  | **44.3**                | 38.1                 |
|              |                     |                      |
| Helpful Base | Best-of-64 (Rating) | Best-of-64 (Ranking) |
| Ranking Eval | 45.1                | **48.8**                 |
| Rating Eval  | **47.0**                | 46.8                 |

- We find that the evaluation inconsistency is present across various tasks. Specifically, the rating evaluation protocol favors the rating best-of-64 policy and vice-versa. This highlights that feedback acquisition protocol biases evaluation.
- We will add these results in the revised paper.

Reviewer comment (Writing):
In Section 2.2 and 2.3, it would be helpful for authors to add references for each to help the reader get a sense of where the protocols and models have been used in prior work.

Author response:
**Q:** Add references in Section 2.2 and 2.3. Some typos, e.g., missing link in the first paragraph of Section 3 and “the humans 6k instances of annotations”.

**A:** We thank the reviewer for their comments on the paper's presentation. We will fix them in the updated version of the paper.

Reviewer comment (Writing):
Some typos, e.g., missing link in the first paragraph of Section 3 and “the humans 6k instances of annotations”.

Author response:
**Q:** Add references in Section 2.2 and 2.3. Some typos, e.g., missing link in the first paragraph of Section 3 and “the humans 6k instances of annotations”.

**A:** We thank the reviewer for their comments on the paper's presentation. We will fix them in the updated version of the paper.

Reviewer comment (Experiments):
This paper does not explore the influence of feedback protocols on common alignment methods (such as RLHF[1], RRHF[2], RLAIF[3], etc.). The alignment in this paper just applies the reward models to select the best out of n generation, which is only affected by the performance of reward models.

Author response:
We thank the reviewer for their thoughtful question. Firstly, we highlight that the best-of-n (rejection sampling) is also a common method in RLHF literature for aligning LLMs. As mentioned in Section 2.3, we choose rejection sampling as due to simplicity, simplicity, and robust performance. For instance, [1] finds that the win-rate of the Best-of-n policy outperforms the base model by 10% absolute points on AlpacaEval.  We do not perform PPO since it requires atleast 8 80GB GPUs according to the AlpacaFarm implementation, which is outside our academic budget. We attempted to train PPO at our compute scale by reducing the effective batch sizes and using LoRA models but it did not run stably. Following the reviewer’s feedback, we performed an additional experiment with rejection sampling finetuning (RSFT) used in three highly cited LLM alignment papers, including LLaMA 2 [2,3,4]. 
Specifically, our setup is as follows:

1. We prompt Alpaca-7B with 5K instructions from Alpaca-52K data.
2. We generate 64 responses for every instruction.
3. We use our rating and ranking reward model to select the best response from the 64 responses.
4. We finetune two Alpaca-7B models with the 5K instruction-responses data. 
(a) One where the responses are chosen from the rating reward model.
(b) Second where the responses are chosen from the ranking reward model.
5. Post finetuning, we sample a single response from the finetuned Alpaca-7B with 553 evaluation instructions
6. We calculate the win-rate against DaVinci003 using the rating and ranking protocol using ChatGPT.

Here are the results for this experiment where baseline is the base Alpaca-7B model:
| Win-rate against Davinci-003 | Baseline | RSFT (Rating) | RSFT (Ranking) |
|------------------------------|----------|---------------|----------------|
| Ranking Evaluation           | 36.9%     | 42.0%          | **43.3%**           |
| Rating Evaluation            | 41.9%    | **44.0%**         | 43.0%           |

We find that the evaluation inconsistency persists under this alignment algorithm too. It indicates that the choice of feedback protocol for the evaluation favors the same feedback protocol used for training the reward models and subsequently finetuning the base LLM. We will add this result in the updated paper. We hope that this experiment answers the reviewer’s comment on the generalizability of the evaluation protocol to other algorithms.

[1] AlpacaFarm: https://arxiv.org/abs/2305.14387 \
[2] RRHF: https://arxiv.org/abs/2304.05302 \
[3] LLaMA2: https://arxiv.org/pdf/2307.09288.pdf \
[4] Constitutional AI: https://arxiv.org/pdf/2212.08073.pdf \
[5] AlpacaFarm Code: https://github.com/tatsu-lab/alpaca_farm#running-reference-methods

Reviewer comment (Evaluation):
The evaluation inconsistency seems straightforward: the performance of reward models will be affected by the format of input data. It is better to convert the rating feedback to the ranking format first and then use it to train an NLL reward model (just like the ranking feedback) and then compare the performance.

Author response:
- In our work, we established that the feedback data obtained from humans and AI suffers from inconsistency problems. We believe that our work is unique as it shows that this has a direct impact on the alignment and evaluation of LLMs through empirical evidence.
- The choice of reward models was naturally made to reflect the nature of the feedback protocol. Specifically, the *rating* protocol provides us with a scalar score hence the regression model makes sense, and the *ranking* protocol does not provide us with scalar scores hence we use a negative log likelihood (NLL) objective function. 
- We observe that 57.4% of the pairwise comparisons are considered “equal” when we convert the rating data into the ranking format (Row 1 of Table 2(a)). In practice, this means that we need to filter almost 60% of the rating data if we want to train a ranking format reward model. This is not the most optimal way to use the rating data when we know that a better regression method exists that can utilize the complete data. Hence, we train a regression reward model in our original setup. 

However, we still perform the experiment as requested by the reviewer. 
1. We train two reward models, of roberta-large architecture, on (a) ranking data with NLL loss and (b) rating data converted to ranking with NLL loss.
2. We perform Best-of-n sampling with n = 64
3. We evaluate the model’s win-rate against davinci-003 with a rating and ranking evaluation scheme. 

We present the results below:

| Win-rate against davinci-003 | Best-of-64 (Ranking) | Best-of-64 (Rating Data Ranking Format) |
|------------------------------|----------------------|-----------------------------------------|
| Relative Eval                | **47.3%**                | 45%                                     |
| Absolute Eval                | 42.0%                | **46%**                                     |

We highlight that the evaluation inconsistency is still evident when the data format used for training the reward models is kept identical. This indicates that our finding is not confounded by the reward model’s data format but the nature of the feedback protocol data itself i.e., rating and ranking. We will update this result in the paper.

Reviewer comment (Experiments):
Can you explore how the feedback protocol affects the reinforcement learning finetuning for model alignment, such as RLHF?

Author response:
We thank the reviewer for their thoughtful question. Firstly, we highlight that the best-of-n (rejection sampling) is also a common method in RLHF literature for aligning LLMs. As mentioned in Section 2.3, we choose rejection sampling as due to simplicity, simplicity, and robust performance. For instance, [1] finds that the win-rate of the Best-of-n policy outperforms the base model by 10% absolute points on AlpacaEval.  We do not perform PPO since it requires atleast 8 80GB GPUs according to the AlpacaFarm implementation, which is outside our academic budget. We attempted to train PPO at our compute scale by reducing the effective batch sizes and using LoRA models but it did not run stably. Following the reviewer’s feedback, we performed an additional experiment with rejection sampling finetuning (RSFT) used in three highly cited LLM alignment papers, including LLaMA 2 [2,3,4]. 
Specifically, our setup is as follows:

1. We prompt Alpaca-7B with 5K instructions from Alpaca-52K data.
2. We generate 64 responses for every instruction.
3. We use our rating and ranking reward model to select the best response from the 64 responses.
4. We finetune two Alpaca-7B models with the 5K instruction-responses data. 
(a) One where the responses are chosen from the rating reward model.
(b) Second where the responses are chosen from the ranking reward model.
5. Post finetuning, we sample a single response from the finetuned Alpaca-7B with 553 evaluation instructions
6. We calculate the win-rate against DaVinci003 using the rating and ranking protocol using ChatGPT.

Here are the results for this experiment where baseline is the base Alpaca-7B model:
| Win-rate against Davinci-003 | Baseline | RSFT (Rating) | RSFT (Ranking) |
|------------------------------|----------|---------------|----------------|
| Ranking Evaluation           | 36.9%     | 42.0%          | **43.3%**           |
| Rating Evaluation            | 41.9%    | **44.0%**         | 43.0%           |

We find that the evaluation inconsistency persists under this alignment algorithm too. It indicates that the choice of feedback protocol for the evaluation favors the same feedback protocol used for training the reward models and subsequently finetuning the base LLM. We will add this result in the updated paper. We hope that this experiment answers the reviewer’s comment on the generalizability of the evaluation protocol to other algorithms.

[1] AlpacaFarm: https://arxiv.org/abs/2305.14387 \
[2] RRHF: https://arxiv.org/abs/2304.05302 \
[3] LLaMA2: https://arxiv.org/pdf/2307.09288.pdf \
[4] Constitutional AI: https://arxiv.org/pdf/2212.08073.pdf \
[5] AlpacaFarm Code: https://github.com/tatsu-lab/alpaca_farm#running-reference-methods

Reviewer comment (Reproducibility):
Will the collected feedback be released?

Author response:
**Q:**  Will the collected feedback be released?

**A:** Yes, we will release the data upon acceptance. To support the claim, we have uploaded the AI feedback data in the supplementary material.

Reviewer comment (Evaluation):
Is there any calibration on the rating scores? For example, detail the meaning of each score (1-7) in the prompt for AI feedback and instruction for human annotation to make sure that the annotators can fully understand the principle of evaluation.

Author response:
**Q:** Is there any calibration on the rating scores?

**A:** 

- We clarify that we collect the rating data on a scale of 1-7 from AI and humans where {1=Very poor, 2=Poor, 3=Below Average,4=Average,5=Above Average,6=Good,7=Excellent}. 
- We provide an annotation guideline in the prompt to the AI explaining the various quality dimensions along which the rating score decision must be based along with two in-context examples – one for low score scenario and second for the high score scenario. In addition, we perform rating data collection for each instance in a new session so that the LLM maintains its same calibration/understanding for the rating scores throughout the annotation process. We will add the prompt to the revised paper.
- We provide the same annotation guidelines as AI to humans. Differently from AI, the humans can (re)calibrate themselves as the annotation progresses. As we ask four workers to annotate the same instance, the small differences caused by the worker’s calibration will be averaged out. Specifically, the feedback data will not be biased towards a specific annotator's calibration, instead reflect the average belief about the response quality under the rating/ranking feedback protocol. We will add the UI screenshot of the human annotator interface in the revised paper.

Reviewer comment (Experiments):
In Section 4.2 on qualitative analysis, the authors sampled a few inconsistent instances and asked annotators to provide explanations for these inconsistencies. This is an important and inconsistent analysis and the annotation protocols should be precisely documented. If the claim “the differences in the preferences of the humans while annotating for different feedback protocols played a significant role in their decision making” is based on only 2–3 instances, the claim may not be fully supported.

Author response:
We thank the reviewer for their insightful question. We agree that it is important to understand the cause of inconsistencies quantitatively and qualitatively. Here, we describe our qualitative analysis for more clarity:

1. In Section 4.2, we show that the feedback data collected from humans and AI is highly inconsistent and analyze their behaviours.
2. Further, we quantify the closeness with the model responses from the lens of rating and ranking feedback protocol. Overall, this analysis suggested that the perceived closeness between the quality of the responses leads to more inconsistent feedback. 
3. To investigate this further, we conduct another small scale experiment. Specifically, we sample a few consistent and inconsistent responses, and then ask the annotators to provide their rating/ranking feedback along with their natural language explanation. We agree with the reviewer that such experiments should be better documented. As a result, we will include the human annotation interface in the updated paper. We believe that our small scale experimental setup is a good design under a finite budget for a qualitative study. 
4. Post-feedback collection, we study the human responses and natural language explanations for both inconsistent (Table 9 and 10) and consistent responses (Table 11 and 12). 
5. We notice that the natural language explanations from humans do explain their choice well. However, the quality aspects that they focus on changes with the feedback protocol. 
6. Specifically we look at the inconsistent feedback example in Table 9. In the ranking setup, the annotators perceived Response 2 as more clever (“making a riddle..”) or correct (“makes more sense..”) over Response 1. Whereas, in the rating setup, the Response 1 gets higher average score as the annotators focussed on the answer’s faithfulness to the task instruction (“Response 2 does not meet the task structure requirements”..)
7. To provide more evidence, we will update the revised paper with more qualitative examples in the Appendix.

Reviewer comment (Novelty):
In the “Alignment and Evaluation” section, the finding that Best-of-n policies outperform SFT has already been reported in prior work, and evaluation inconsistency is predictable given the discussion of inconsistencies. While the second part (inconsistencies) could be novel, the descriptions of the results are confusing.

Author response:
We thank the reviewer for raising their concern. We describe our process here for more clarity:

1. In Section 4, we show that the feedback protocols (rating/ranking) do not agree with each other i.e., 60% disagreements between the humans and AI as annotators.
2. This begs the question: how does the feedback inconsistency affect our aligned LLMs and their evaluation. 
3. To understand this end, we train reward models on the rating and ranking data independently.
4. We prompt the base Alpaca model to generate 64 responses for the 553 unseen instructions taken from AlpacaEval [1] dataset.
5. We re-rank the 64 responses using the rating and relative reward model independent of each other, and choose the best response.
6. Finally, we show the responses from this best-of-n strategy and a reference model (davinci-003) to the annotators.
7. We ask a different set of human annotators to provide their rating and ranking feedback on the responses. In addition, we also ask ChatGPT as an annotator to provide their rating and ranking feedback on the responses. We clarify that the rating feedback is collected independently for each response whereas the ranking feedback is a pairwise evaluation.
8. Finally, we calculate the fraction of times where the response from best-of-n strategy using rating/ranking reward model is preferred over the davinci-003 response.
9. For the ranking evaluation, we compare if the annotators chose responses from best-of-n or davinci-003. For the rating evaluation, we compare the ratings given to the responses from best-of-n and davinci-003, and compare them. This gives us the win-rates for rating and ranking strategies using human and AI annotators.
10. In Figure 3, we show that the feedback protocol chosen during evaluation prefers the responses from the best-of-n policy that utilizes the same feedback protocol. That is, ranking evaluation prefers the responses from the ranking reward model and vice-versa.
11. Takeaway: Preference acquisition strategy biases evaluation. 
12. Our work thus highlights that the impact of different design choices can be drastic for the LLM alignment pipeline.

We believe that our observation is novel and has not been reported in the prior work. We will be happy to provide further clarifications if there are further concerns.
[1] AlpacaEval: https://github.com/tatsu-lab/alpaca_eval

Reviewer comment (Experiments):
The reviewer’s understanding is that if a ranking model is used for Best-of-n ranking, higher rates occur when the same ranking protocol is used during evaluations. This is seen as unsurprising because the reward model is trained on the feedback data and re-ranks n responses at inference time, meaning that if it is trained on pairwise feedback data, it learns to choose the response preferred in the pairwise setup. It would be interesting to see if using the feedback data during training (e.g., PPO) would yield the same trends. The reviewer remains confused by the descriptions in these paragraphs.

Author response:
We thank the reviewer for their thoughtful question. Firstly, we highlight that the best-of-n (rejection sampling) is also a common method in RLHF literature for aligning LLMs. As mentioned in Section 2.3, we choose rejection sampling as due to simplicity, simplicity, and robust performance. For instance, [1] finds that the win-rate of the Best-of-n policy outperforms the base model by 10% absolute points on AlpacaEval.  We do not perform PPO since it requires atleast 8 80GB GPUs according to the AlpacaFarm implementation, which is outside our academic budget. We attempted to train PPO at our compute scale by reducing the effective batch sizes and using LoRA models but it did not run stably. Following the reviewer’s feedback, we performed an additional experiment with rejection sampling finetuning (RSFT) used in three highly cited LLM alignment papers, including LLaMA 2 [2,3,4]. 
Specifically, our setup is as follows:

1. We prompt Alpaca-7B with 5K instructions from Alpaca-52K data.
2. We generate 64 responses for every instruction.
3. We use our rating and ranking reward model to select the best response from the 64 responses.
4. We finetune two Alpaca-7B models with the 5K instruction-responses data. 
(a) One where the responses are chosen from the rating reward model.
(b) Second where the responses are chosen from the ranking reward model.
5. Post finetuning, we sample a single response from the finetuned Alpaca-7B with 553 evaluation instructions
6. We calculate the win-rate against DaVinci003 using the rating and ranking protocol using ChatGPT.

Here are the results for this experiment where baseline is the base Alpaca-7B model:
| Win-rate against Davinci-003 | Baseline | RSFT (Rating) | RSFT (Ranking) |
|------------------------------|----------|---------------|----------------|
| Ranking Evaluation           | 36.9%     | 42.0%          | **43.3%**           |
| Rating Evaluation            | 41.9%    | **44.0%**         | 43.0%           |

We find that the evaluation inconsistency persists under this alignment algorithm too. It indicates that the choice of feedback protocol for the evaluation favors the same feedback protocol used for training the reward models and subsequently finetuning the base LLM. We will add this result in the updated paper. We hope that this experiment answers the reviewer’s comment on the generalizability of the evaluation protocol to other algorithms.

[1] AlpacaFarm: https://arxiv.org/abs/2305.14387 \
[2] RRHF: https://arxiv.org/abs/2304.05302 \
[3] LLaMA2: https://arxiv.org/pdf/2307.09288.pdf \
[4] Constitutional AI: https://arxiv.org/pdf/2212.08073.pdf \
[5] AlpacaFarm Code: https://github.com/tatsu-lab/alpaca_farm#running-reference-methods

Reviewer comment (Reproducibility):
Could you provide the details of the human annotation process of qualitative analysis?

Author response:
- We clarify that this experiment is only performed with the human annotators. The group of annotators that provide rating/ranking with explanations is mutually exclusive.
- In total, we consider a subset of 50 instructions and 100 responses (2 responses per instruction) from our dataset. In addition, we assign 3 annotators per rating/ranking instance.
- The humans are provided the same annotation guidelines as before but they are asked to provide an explanation for their annotation too. We provide them 2 solved examples of how the explanations could look like. 
- We intended to keep the human explanations free from any biases from the authors, hence, we instructed the humans to provide their explanation based on their perception of the response quality. We will add the UI screenshot of the human annotator interface for rating and ranking qualitative assessment in the revised paper.
- We present a few examples of the human explanations in Table 7-10 of the paper. We will add a couple of more examples in the revised version of the paper.

Reviewer comment (Experiments):
In Table 2, the authors showed that when converting the rating to ranking, there is a huge disagreement between the two different protocols. One flaw is that a proper baseline is missing here. For example, if you annotate the same examples using another batch of human or sample from the same model (GPT-3.5-Turbo) using different temperature, what would be the disagreement? I would be curious to see such a baseline as it also correlates with the main conclusion of this paper.

Author response:
**Q:** Annotate the same examples using another batch of human or sample from the same model (GPT-3.5-Turbo) using different temperature, what would be the disagreement? 

**A:** We thank the reviewer for their insightful question. Firstly, we provide further clarifications on collecting the feedback data from the humans.

1. During the feedback acquisition, we create two exclusive groups of humans – one that provides the rating feedback and ranking feedback.
2. For the rating protocol, four humans rate an individual response for a given instruction. 
3. Similarly, four humans rank a pair of responses for a given instruction under the ranking protocol.
4. As described in the Agreement Analysis (3.1), we consider three annotations as the predicted human preferences and the fourth annotation as the human gold label. 
5. Table 2(b) indicates consistency analysis of the predicted human preferences i.e., randomly selecting 3 annotators out of 4 annotators. 
6. To provide further analysis as requested by the reviewer, we calculate the consistency evaluation for a different batch of randomly selected 3 annotators four times. 
7. We find that the standard deviation of the inconsistency analysis for four splits of the humans evaluators is **1.6%**. This indicates that the inconsistency analysis is robust to the batch of humans used for annotation. 

To provide more evidence for the same model using a different temperatures – 0, 0.2 and 0.5 for 1000 comparisons. We find that the standard deviation in the disagreements between these temperatures is **2.7%**.  This indicates that the inconsistency analysis is also robust to the different model temperatures used for generating feedback data. We will add these statistics in our updated paper.

Reviewer comment (Experiments):
This is not necessarily a weakness but given what observed in the paper, it would be great to see if the authors could take one step further to verify the root cause of the difference and try potential ways to combine the advantages of the two annotation protocols. To me, Figure 3 shows that the ranking-based protocol is more effective and robust (it yield better win rate with ranking-based evaluation protocol and smaller gap when changing the evaluation protocol). This makes sense, as in the rating protocol, the model/human only see each individual answer thus the rating is likely to be less calibrated. Also, seen from the examples in section H, the ranking protocol is likely to introduce some systematic biases. Here is my proposal: can we try the third protocol that instruct the model/human to give both ratings and rankings for either a pair of responses or a list of responses. My guess is that rating annotated in such a way will be better calibrated and the rankings will less likely to be biased.

Author response:
**Q: Root cause of the difference**

**A:**

We agree that the root cause of the difference between the feedback protocols is quite important. Towards this, we highlight our quantitative and qualitative findings in the Section 4.2:

1. We show that the feedback data collected from humans and AI is highly inconsistent and analyze their behaviours.
2. Further, we  quantify the closeness with the model responses from the lens of rating and ranking feedback protocol. Overall, this analysis suggested that the perceived closeness between the quality of the responses leads to more inconsistent feedback. 
3. To investigate this further, we conduct another small scale experiment. Specifically, we sample a few consistent and inconsistent responses, and then ask the annotators to provide their rating/ranking feedback along with their natural language explanation. We agree with the reviewer that such experiments should be better documented. As a result, we will include the human annotation interface in the updated paper. We believe that our small scale experimental setup is a good design under a finite budget for a qualitative study. 
4. Post-feedback collection, we study the human responses and natural language explanations for both inconsistent (Table 9 and 10) and consistent responses (Table 11 and 12). 
5. We notice that the natural language explanations from humans do explain their choice well. However, the quality aspects that they focus on changes with the feedback protocol. 
6. Specifically we look at the inconsistent feedback example in Table 9. In the ranking setup, the annotators perceived Response 2 as more clever (“making a riddle..”) or correct (“makes more sense..”) over Response 1. Whereas, in the rating setup, the Response 1 gets higher average score as the annotators focussed on the answer’s faithfulness to the task instruction (“Response 2 does not meet the task structure requirements”..)
7. To provide more evidence, we will update the revised paper with more qualitative examples in the Appendix.