Paper: Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
The main contribution of this paper is the FEEDBACK COLLECTION dataset. However, the dataset has not been made publicly available, and the construction details are unclear. For instance, the content of Step 2 is incomplete, and Step 3 is overly simplistic. Furthermore, the prompts used during construction have not been disclosed.

Author response:
### **(W1) Dataset Publicity Issues and Construction Details of the Feedback Collection**

#### [**Dataset Publicity Issues**]
Regarding our dataset, we couldn’t upload our file due to its large size (OpenReview has a 100MB limit) and we didn’t include the link due to the anonymous policy (it is currently open-source at a public website). We will include the link in our camera-ready version. Also, we have uploaded our anonymized version of the code as supplementary material. 

#### [**Details for Dataset Construction Process**]

Here is a detailed explanation of the dataset construction process in the updated draft (Section 3.1) :

The collection process consists of (1) the curation of 50 initial seed rubrics, (2) the expansion of 1K new score rubrics through GPT-4, (3) the augmentation of realistic instructions, and (4) the augmentation of the remaining components in the training instances (i.e. responses including the reference answers, feedback, and scores). Figure 6 shows the overall augmentation process.

* Step 1: Creation of the Seed Rubrics We begin with the creation of a foundational seed dataset
of scoring rubrics. Each author curates a detailed and fine-grained scoring rubric that each personnel considers pivotal in evaluating outputs from LLMs. This results in an initial batch of 50 seed rubrics.

* Step 2: Augmenting the Seed Rubrics with GPT-4 Using GPT-4 and our initial seed rubrics, we
expand the score rubrics from the initial 50 to a more robust and diverse set of 1000 score rubrics. Specifically, by sampling 4 random score rubrics from the initial seed, we use them as demonstrations for in-context learning (ICL), and prompt GPT-4 to brainstorm a new novel score rubric. Also, we prompt GPT-4 to paraphrase the newly generated rubrics in order to ensure PROMETHEUS could generalize to the similar score rubric that uses different words. We iterate the brainstorming → paraphrasing process for 10 rounds. The detailed prompt used for this procedure is in Appendix J.

* Step 3: Crafting Novel Instructions related to the Score Rubrics With a comprehensive dataset of 1000 rubrics at our disposal, the subsequent challenge was to craft pertinent training instances. For example, a score rubric asking “Is it formal enough to send to my boss” is not related to a math problem. Considering the need for a set of instructions closely related to the score rubrics, we prompt GPT-4 to generate 20K unique instructions that are highly relevant to the given score rubric.

* Step 4: Crafting Training Instances Lastly, we sequentially generate a response to evaluate and corresponding feedback by prompting GPT-4 to generate each component that will get a score of i (1 ≤ i ≤ 5). This leads to 20 instructions for each score rubric, and 5 responses & feedback for each instruction. To eliminate the effect of decision bias when fine-tuning our evaluator LM, we generate an equal number of 20K responses for each score. Note that for the response with a score of 5, we generated two distinctive responses so we could use one of them as an input (reference answer).

For the prompts used for each step, please refer to Appendix J in the updated draft. We hope the updated version will waive your concern regarding the lack of details.

Reviewer comment (Experiments):
It is unclear whether the test data in Table 1 is manually constructed or generated by GPT4-0613. If it is generated by GPT4-0613, why are the Pearson/Kendall/Spearman evaluation metrics not equal to 1?

Author response:
### **(W3) GPT-4’s discrepancy between Generating the Data / Solving it again**
The test dataset of the Feedback Collection was constructed using GPT-4-0613. The detailed process is identical to how the training set was constructed. Yet, the difference is that we ensured that the scoring criteria does not overlap with the training set as shown in Figure 7 (Left skewed Rouge Score distribution).

We conjecture that the reason why GPT-4-0613 doesn’t acquire a correlation of 1.0 is that there is a **discrepancy between generating a new instruction & evaluating the problem it generated**. On the other hand, Prometheus was specifically trained on such instruction, hence we conjecture that the **distribution is shifted narrowly towards the Feedback Collection** that GPT-4-0613 generated. 

We have included this information in Appendix C.3.

Reviewer comment (Evaluation):
For the Unseen FEEDBACK COLLECTION Testset, should all unseen instances be extracted? If the evaluation is conducted by combining the 50 Unseen samples with the 1000 samples similar to the training distribution, would this overshadow the true performance when facing unseen distribution during training?

Author response:
### **(W4) Explanation of the Feedback Collection Test Set**
We have not fully understood the point of this question, hence we would kindly like to request whether you could elaborate on it.

For reference, the following is a more detailed explanation of how the seen and unseen subset within the test set differs. 

#### **[Difference Between the Seen (Rubric) Subset and the Unseen (Rubric) Subset**

* Both the seen and unseen subset contains unseen instructions and responses compared to the training set. The difference is whether the score rubric was seen during training.

* Seen (Rubric) Test set: This subset contains the exact 1K score rubrics within the training data. Yet, it has 1K novel instructions and responses that are not included in the training data.

* Unseen (Rubric) Test set: This subset contains 50 novel score rubrics that are not contained within the training data. It also contains 1K instructions and responses that are not included in the training data.

* Since both the Seen (Rubric) and Unseen (Rubric) have **non-overlapping instructions and responses** compared to the training data, we argue that this is not problematic during the evaluation process of Prometheus.

We have included this information in Appendix B.1 in the updated draft.

Reviewer comment (Experiments):
One of my concerns about this work is whether can Prometheus be generalized to other fields since the downstream benchmarks are close the the training data. More results on unseen data and more specific domains can better improve this work.

Author response:
### **(W1) Generalization to Other Fields**

#### [**Difference Between the Feedback Collection and the Datasets used for Evaluation**]
We would first like to emphasize that the datasets used for experiments in Table 3 (Vicuna Bench, MT Bench, Flask) contains instructions that have different characteristics with the training dataset we used (Feedback Collection).

Specifically, the dataset in Table 4 have the following characteristics:

* **Vicuna Bench**:Relatively short instructions that are mostly consisted of How/What type of questions in addition to some coding and math instructions. (e.g., "How can governments utilize fiscal and monetary policies to combat economic recessions?)

* **MT Bench**: The 2nd instruction we used in this multi-turn dataset mostly asks for revision or further explanation of the previous response. (e.g., “Now, do the same task again but only use four-word sentences.”)

* **Flask Eval**: Contains academic NLP tasks such as classification datasets (e.g., MMLU, FEVER) or the train split of instruction datasets (e.g., Self-Instruct, WizardLM) (e.g., "Paraphrase the given text in an academic style. Input: Lots of papers have been published on this topic.”)

In contrast, the **Feedback Collection** was constructed with the main consideration of a very detailed, realistic situation where a user is interacting with an AI. (e.g., “Suppose there is a friend who is feeling low due to a poor performance on a recent exam. The friend is now seeking advice, encouragement, and a bit of humor to lighten the mood. How would one approach this situation, incorporating wit and a playful tone to uplift the friend's spirit?”)

The point we would like to emphasize is that Prometheus shows a higher correlation with both human evaluators (Figure 3) and GPT-4 (Table 3) in all of these different datasets. This supports that Prometheus could generalize to other instruction evaluation settings.

#### [**Prometheus also works in Coarse-grained Relative Evaluation Setting although it was trained for Fine-grained Aboslute Evaluation**]

Also, we would like to emphasize that the human preference dataset experiments in Table 4 (HHH Alignment, MT Bench Human Judgment) is evidence indicating the generalizability of Prometheus.

Specifically, we would like to emphasize the following two points:

* Results show that Prometheus could **generalize to coarse-grained criteria** (Helpfulness, Harmlessness, Honesty) although it was trained with detailed and fine-grained criteria (Cultural Sensitivity, Humorous, Considering regulation and compliance requirements).

* Results show that while Prometheus was trained in an absolute scoring setting, its evaluation capabilities could also be **transferred to a relative scoring setting**. Prometheus outperforms open-source reward models that were specifically trained with human preference datasets (StanfordNLP Reward Model, ALMOST) and even GPT-3.5-Turbo.  Although it doesn’t get access to both responses, it could manage to make a score decision that gives a higher score to human-preferred responses. Without any generalization, this would be extremely hard.

Reviewer comment (Evaluation):
Potential bias of Prometheus. Can Prometheus be attacked by some adversarial attack methods? Does it have stronger biases like length bias compared with GPT-4?

Author response:
### **(W2) Potential bias of Prometheus**

#### [**Absolute Scoring is less vulnerable to Length Bias**]
We strongly agree that your concern regarding various biases that could occur during evaluation is very crucial in the field. As you specifically mentioned about **length bias**, we would like to discuss more about it in detail.

We would like to highlight that in this aspect, evaluation based on **Absolute Scoring holds a strong advantage over Relative Scoring** since the evaluator LM doesn’t get vulnerable to the length differences or other surface patterns between the two responses during evaluation. 

In Figures 13,14 and 15, we have measured whether Prometheus or GPT-4 gave a higher score to longer responses in an absolute evaluation setting and found that it is not the case. The responses that got a score of 1 - 5 all have similar score distributions, supporting the idea that Absolute Scoring might be a good way to mitigate length biases.

#### [**Additional Experiment on Adversarial Dataset**]

To further analyze other datasets, we have conducted experiments with the recently proposed LLMBar dataset [1]. This dataset has an adversarial subset where the better response is shorter and the worse response is longer. With the same setting as in our human preference datasets in Table 4, we have conducted additional experiments and obtained the following results:

|          | Random Guess | Llama-2-Chat (70B) | GPT-3.5-Turbo-0613 | Prometheus-13B | GPT-4-0613 |
|----------|--------------|--------------------|--------------------|----------------|------------|
| Accuracy | 48.91        | 35.87              | 44.57              | 63.04          | 75.00      |

This result supports that Prometheus is relatively robust to adversarial patterns such as length bias compared to other baselines such as Llama-2-Chat (70B) and GPT-3.5-Turbo-0613.

Reviewer comment (Reproducibility):
Model implementation is not described.

Author response:
### **(W1) Details of Model Implementation**

Our explanation of the model implementation is included in Section 3.2 and Appendix F. Using the llama-recipes repository, we have fine-tuned Llama-2-Chat (7B & 13B) to obtain Prometheus using 8 A100 (80GB) GPUs with FSDP. The training is similar to Chain-of-Thought Fine-tuning, where we fine-tune to sequentially generate the feedback and the the score.

The hyperparameters we used during training and inference are included in Table 8 and 9.

Reviewer comment (Experiments):
Experimental methodology not clear or supported.

Author response:
### **(W2) Explanation of the Experimental Methodology**

Since our main objective is to obtain a open-source evaluator language model that could closely emulate GPT-4 and human evaluators, we mainly divide our experiments into two different phases. Further details are included in Appendix B.

* We first check how the score is similar with either human evaluators (as shown in Figure 3) and GPT-4 (as shown in Table 2 and 3). Specifically, after parsing the score decision, we compare with Pearson, Spearman, and Kendall-Tau correlation with the opponent.
* Next we check how the other component that Prometheus generated, which is the language feedback has good quality. This is shown in Figure 4 along with an analysis in Figure 5. Specifically, we ask human evaluators to compare the language feedback and choose among Prometheus and GPT-4.

Reviewer comment (Presentation):
Most figures missing.

Author response:
### **(W3) Explanation of our Contribution**

The main contributions of our work are as follows:

* We introduce the FEEDBACK COLLECTION dataset specifically designed to train an evaluator LM. Compared to previous feedback datasets, it includes customized scoring rubrics and reference answers in addition to the instructions, responses, and feedback.
* We train PROMETHEUS, the first open-source LLM specialized for fine-grained evaluation that can generalize to diverse, real-world scoring rubrics beyond a single-dimensional preference such as helpfulness and harmlessness.
* We conduct extensive experiments showing that by appending reference materials (reference answers, fine-grained score rubrics) and fine-tuning on feedback, we can induce evaluation capability into language models. PROMETHEUS shows high correlation with human evaluation, GPT-4 evaluation in absolute scoring settings, and also shows high accuracy in
ranking scoring settings.

Reviewer comment (Novelty):
Contribution too small (not any new data, model or any advertised contribution is clearly described).

Author response:
### **(W3) Explanation of our Contribution**

The main contributions of our work are as follows:

* We introduce the FEEDBACK COLLECTION dataset specifically designed to train an evaluator LM. Compared to previous feedback datasets, it includes customized scoring rubrics and reference answers in addition to the instructions, responses, and feedback.
* We train PROMETHEUS, the first open-source LLM specialized for fine-grained evaluation that can generalize to diverse, real-world scoring rubrics beyond a single-dimensional preference such as helpfulness and harmlessness.
* We conduct extensive experiments showing that by appending reference materials (reference answers, fine-grained score rubrics) and fine-tuning on feedback, we can induce evaluation capability into language models. PROMETHEUS shows high correlation with human evaluation, GPT-4 evaluation in absolute scoring settings, and also shows high accuracy in
ranking scoring settings.

Reviewer comment (Experiments):
Data is synthetic and not corrected by humans for any potential errors.

Author response:
### **(W4) Explanation for the Synthetic Nature of the Data**

#### [**Prometheus shows high correlation with human evaluators**]

Your concern regarding the synthetic nature of the Feedback Collection is valid. This was also one of our main concerns, which is why we conducted correlation with human evaluators in the MT Bench, VicunaBench, and Feedback Collection test set (shown in Figure 3). In addition to the score decision, human evaluators considered the feedback generated by Prometheus to be as valid as GPT-4 (shown in Figure 4).

In addition, we have conducted experiments with 200 instances from the Flask Eval dataset to measure the correlation with human evaluators (an extension from Table 3).

| Models             | Pearson Correlation w/ Human Evaluators | Pearson Correlation w/ GPT-4-0613 |
|--------------------|-----------------------------------------|-----------------------------------|
| Llama-2-Chat-7B    | 0.298                                   | 0.271                             |
| Llama-2-Chat-13B   | 0.275                                   | 0.265                             |
| Llama-2-Chat-70B   | 0.317                                   | 0.336                             |
| Prometheus-7B      | 0.327                                   | 0.367                             |
| Prometheus-13B     | **0.451**                               | **0.467**                         |
| GPT-3.5-Turbo-0613 | 0.421                                   | 0.422                             |
| GPT-4-0613         | 0.729                                   | 0.835 (Self-Consistency)          |

This results support that although Prometheus was trained on a Synthetic dataset (Feedback Collection), it shows a higher correlation with human evaluators compared to other baselines such as GPT-3.5-Turbo and Llama-2-Chat (70B).

#### [**Prohibitive Costs**]

We would also like to emphasize that although manually inspecting all of the instances might be the best way to correct potential errors, this would be extremely expensive to do. We approximately spent $8,000 to create the Feedback Collection, collecting or revising the same amount with only humans will cost at least 10x, especially because the level of expertise required for such annotations is very high.

#### [**Related Works also use GPT-4 to augment training data**]

Lastly, we would like to note that a lot of recent work [1,2,3,4] heavily relies on datasets augmented from Proprietary LLMs as it is cost-efficient and as effective as obtaining good quality data compared to human annotation.

---

### References
[1] Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D. and Hajishirzi, H., 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.

[2] Peng, B., Li, C., He, P., Galley, M. and Gao, J., 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.

[3] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P. and Hashimoto, T.B., 2023. Stanford alpaca: An instruction-following llama model.

[4] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T., 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.

Reviewer comment (Presentation):
Where is Figure 2?

Author response:
Before addressing the other issues, I think the problem of the missing figure might be due to your browser.
I had the same problem before when using Safari. The draft size is quite large and Safari doesn't seem to render it very well.

Could you try another browser or environment to check if the problem holds?
Currently, Figure 2 and Figure 4 are included in the draft.

Reviewer comment (Presentation):
Where is Figure 4?

Author response:
Before addressing the other issues, I think the problem of the missing figure might be due to your browser.
I had the same problem before when using Safari. The draft size is quite large and Safari doesn't seem to render it very well.

Could you try another browser or environment to check if the problem holds?
Currently, Figure 2 and Figure 4 are included in the draft.