Paper: UltraFeedback: Boosting Language Models with High-quality Feedback (ICLR-2024)

Reviewer comment (Experiments):
Are all the data annotated by GPT4? Is there a part of the data annotated by humans?

Author response:
A2: Yes, the released data are all annotated by GPT-4. We selected 100 samples for human annotation and discussed their alignment issue in Section 5.1.

Reviewer comment (Experiments):
The instructions are all single turns? Does the dataset support the training of multi-turn scenarios?

Author response:
A3: Yes, all instructions are single turns due to the budget limitation. To resolve this issue, we mixed UltraFeedback and other datasets like HH-RLHF and OASSIST1 to train UltraRM. As the experimental results indicate, UltraRM performs pretty good at multi-turn reward modeling on HH-RLHF.

Reviewer comment (Evaluation):
Lack of analysis on critique: Given the diversity and complexity of the tasks in ULTRAFEEDBACK, the critique could be task-dependent and complex. More analysis should be presented to help readers to better understand the properties of the critique. The same applies to UltraCM - we only know it generates good critiques but are not sure how useful these critiques are.

Author response:
A2: Thanks for the valuable advice. To comprehensively evaluate and analyze the critiques our model generates, we conducted case study in Appendix E.4. On math problem benchmarks, we found that UltraCM-13B could give meaningful critiques on incorrect answers, pointing out flaws and providing suggestions. However, the suggestions are sometimes unnecessarily long, which indicates the imperfection of the model. We will continue to improve the critique quality in future work.

Reviewer comment (Novelty):
The contribution of this paper is mainly the proposed dataset, which is collected by widely using the GPT4 API. The method part is all standard method. The collected dataset can be used for research purposes but commercial use is illegal. If ICLR sees a pure dataset contribution paper as appropriate for this conference, then I am ok with accepting it. I am going to give a "weak reject" first and then will check and discuss with AC to see whether I will change my score.

Author response:
Thank you for the valuable comment! 

> Q1: If ICLR sees a pure dataset contribution paper is appropriate for this conference, then I am ok with accepting it. I am going to give a "weak reject" first and then will check and discuss with AC to see whether I will change my score.

A1: As shown in the official ICLR 2024 [website](https://iclr.cc/Conferences/2024/CallForPapers), there is a subject area named *datasets and benchmarks*, which is also exactly the primary area of our paper (Please see above). Therefore, we believe dataset papers are ok for ICLR.

Reviewer comment (Evaluation):
From Table 2, by comparing the "UltraRM-UF" and "UltraRM-Overall", it looks to me that "UltraRM-Overall" significantly outperforms "UltraRM-UF" in at least 3 out of 4 benchmark test sets. However, this paper claims that fine-grained scores outperform the overall score. For example, the claim "UltraRMOverall discernably lags behind UltraRM-UF and UltraRM on WebGPT. There can be two potential explanations for this observation. First, fine-grained annotation, which scores model outputs from different aspects respectively, provides a more precise assessment for each completion than aggregating evaluation into an overall number." This claim is controversial given the provided results.

Author response:
> Q4: From Table 2, by comparing the "UltraRM-UF" and "UltraRM-Overall", it looks to me that "UltraRM-Overall" significantly outperforms "UltraRM-UF" in at least 3 out of 4 benchmark test sets, however, this paper claims that fine-grained scores outperform the overall score. For example of this claim "UltraRMOverall discernably lags behind UltraRM-UF and UltraRM on WebGPT. There can be two potential explanations for this observation. First, fine-grained annotation, which scores model outputs from different aspects respectively, provides a more precise assessment for each completion than aggregating evaluation into an overall number.". This claim is controversial given the provided results.

A4: Sorry for the confusion. Actually, "UltraRM-UF" refers to the model solely trained on UltraFeedback with fine-grained scores, and "UltraRM-Overall" is trained on UltraFeedback mixed with open-source datasets with overall scores (see Section 3.4). We list the three different versions of UltraRM as below. Therefore, comparing "UltraRM-Overall" and "UltraRM" is the right way to assess the effect of fine-grained scores. We will further clarify the difference in the revised version.

|                     | Data                      | Score        |
| ------------------- | ------------------------- | ------------ |
| **UltraRM-UF**      | UltraFeedback             | Fine-grained |
| **UltraRM-Overall** | UltraFeedback+Open-Source | Overall      |
| **UltraRM**         | UltraFeedback+Open-Source | Fine-grained |

References

[1] Wang et al. OpenChat: Advancing Open-source Language Models with Mixed-Quality Data. 2023

[2] Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023.

[3] Zheng et al. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. 2023.

Reviewer comment (Evaluation):
In Table 2, for the StreamSHP model and for the SHP dataset, from this link: https://huggingface.co/datasets/stanfordnlp/SHP, the model should achieve 72.8% on the SHP test set. However, the performance in this work is reported as "51.6%". Could you explain the huge gap here?

Author response:
> Q2: In Table 2, for the StreamSHP model and for the SHP dataset, from this link: https://huggingface.co/datasets/stanfordnlp/SHP, the model should achieve 72.8% on the SHP test set, however, the performance in this work reports as "51.6%". Could you explain the huge gap here?

A2: We are sorry for the ambiguity. The SteamSHP model is different from other reward models in the input format, for it accepts two responses simultaneously (see the model [page](https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-xl)) and outputs which one is better (text-to-text format). During the experiment, we found that there is a position bias issue for this approach, where the reward model tends to prefer the first responses. To eliminate this issue, we average the scores from two runs exchanging response orders to get the final scores, leading to inconsistency. We will add a section in appendix for this issue and present the scores for both runs in the revised paper.