Paper: PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations (ICLR-2024)

Reviewer comment (Experiments):
The study seems to be limited to question answering only. LLM has many other applications such as writing reviews and generating essays. Would the proposed methods be applicable to improve evaluations on those applications as well?

Author response:
- We kindly point out that we evaluated our model on the diverse Vicuna80 dataset encompassing a wide variety of tasks: writing, roleplay, common sense, fermi, counterfactual, coding, math, generic, knowledge, etc. Each task includes over 10 questions.

Reviewer comment (Evaluation):
How significant are the result comparisons, such as in Table 3, 4 and 5? It'd be great if the authors could present t-test results such as p-values of the comparisons.

Author response:
- We added significance tests for all of the results as shown below:
- Table 3 shows the example-level correlation results for peer rank. Two-tailed t-test results are shown on the accuracy measure for our method compared to the other baselines (statistical significance is indicated with \*(p<0.01), \*\*(p<0.002))

| Reviewer                 | Fleiss Kappa | Accuracy |
|--------------------------|--------------|----------|
| GPT-3.5                  | 0.387        | 0.621    |
| Claude                   | 0.319        | 0.607    |
| GPT-4                    | 0.406        | 0.643    |
| GPT-4 & Claude & GPT-3.5 | 0.403        | 0.666*   |
| All Reviewers (Weighted) | 0.410        | 0.673**  |

- Table 4 describes peer discussion accuracy on LFQA. We note that when PD happens between a model and a different model (e.g. Claude & GPT-4), results are significantly better. Statistical significance is indicated with \*(p<0.05), \*\*(p<0.005). 

|                     | R1    | R2    | R1 lead | R2 lead | Random |
|---------------------|-------|-------|---------|---------|--------|
| GPT4 & Claude       | 0.729 | 0.671 |  0.743* |  0.729  |  0.729 |
| GPT4 & GPT35        | 0.729 | 0.579 |  0.714  |  0.750* |  0.731 |
| GPT35 & Claude      | 0.579 | 0.671 | 0.700** |  0.671  |  0.686 |
| GPT35 & GPT35-0.8   | 0.579 | 0.650 |  0.664  | 0.686** |  0.681 |
| Claude & Claude-0.8 | 0.664 | 0.707 |  0.693  |  0.671  |  0.680 |
| GTP4 & GPT4-0.8     | 0.779 | 0.757 |  0.779  |  0.757  |  0.779 |

- Table 5 shows the comparison of PD and PR on Vicuna80. Two-tailed t-test results are shown on the F1 measure for our method compared to the baseline.

| Reviewer       | Accuracy         |
|----------------|------------------|
| GPT-4          | 0.3500           |
| GPT-4 & Claude | 0.3675 (p<0.01)  |
| All            | 0.4375 (p<0.005) |
| All (weighted) | 0.4625 (p<0.002) |

Reviewer comment (Experiments):
Is there any data contamination issues for the LLMs being tested?

Author response:
- The models tested are released before the creation of our Vicuna80 and LFQA datasets, so there should be no or little contamination happening.
- Besides, there are no golden answers to open-ended questions in our datasets and there are possibly large sets of good answers. Thus, there are much fewer data contamination issues as compared to exam-like questions [[1](https://arxiv.org/abs/1911.07176), [2](https://arxiv.org/abs/2009.03300)].

Reviewer comment (Writing):
I don't feel equipped to evaluate its conceptual weaknesses because I cannot tell what the motivation, the problem, the proposed solution, or the result is.

Author response:
We thank the reviewer for the feedback.
# Weaknesses
> 1. **Motivation and problem**
- We focus on the topic of utilizing large language models (LLMs) for evaluating open-ended generation instead of proposing a decoding strategy. The goal is to propose new methods using LLMs for better and fair evaluations that align with human judgments. 
	- The basic task formulation is like, given answers A and B, the model is required to evaluate which of them is better.
	- Determining pairwise comparison is also a key step for InstructGPT training [7].
- To get an overview and basic knowledge of the topic, we kindly refer the reviewer to skim a set of related work on LLM-based evaluation [1,2,3,4,5,6,7], where [1] is a very basic model in this direction.

> 2. **Proposed solution**
- We propose (1) the peer rank (PR) algorithm that takes into account each peer LLM’s pairwise preferences of all answer pairs and outputs a final ranking of model capabilities;
- And (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on preferences of two answers.

> 3. **Results**
- We conduct comprehensive experiments on two benchmark datasets. We find that our approaches achieve significantly higher accuracy and better alignment with human preference.

Reviewer comment (Novelty):
Is the aim of your method to help LMs produce better answers to questions than other methods currently achieve?

Author response:
# Questions
> 1. **Is the aim of your method to help LMs produce better answers to questions than other methods currently achieve?**
- As outlined in the abstract and introduction, our aim is to **introduce improved evaluation methods for large language models (LLMs)** that enhance accuracy and better align with human judgments. Our research opens up possibilities for assessing models that are challenging for humans to compare.

Reviewer comment (Evaluation):
I see limited improvements when PR and PD are applied. In Table 2, the Elo rating becomes closer to human raters' rating, but the ranking of the five LLMs does not change. In Table 3, the example-correlation does not increase too much, comparing "All Reviewers (Weighted)" to "GPT-4 & Claude & GPT-3.5". These indicate that the core idea of PR may not be very useful. In Table 5, the performance boost with PD is not that great.

Author response:
> 1. **Limited improvements PR. Elo rating becomes closer to human raters' rating, but the ranking of the five LLMs does not change.**
- We would politely point out that the Elo score is a more fine-grained and important metric for comparing model capabilities. Recent works [1, 2, 3] all used it as the main metric. This metric is important even if the ranking doesn’t change in the specific case. The Elo rating of "All (Weighted)" is the closest to human rating. It shows our method has a better alignment with human judgment.
- Ranking is more general and coarse-grained and is not informative when the gap of model capabilities is large. Thus it’s more convincing to review the rank when including models that are of similar capabilities (on a challenging set of questions). –  we added new experiments including another set of LLMs.

**Additional Experiment:**
- In the new experiments, we added new public LLM and conducted PR among Vicuna-13b, Zephyr-7b-alpha, and GPT-3.5. **We show that both ranking and Elo change and become closer to the golden ranking**, which is consistent with the experiments of commercial LLMs. This change demonstrates PR also contributes to adjusting the global ranking.

|         |  All |      | All (Weighted) |      | Human Raters |      |
|---------|:----:|:----:|:--------------:|:----:|:------------:|:----:|
| Models  |  Elo | Rank |       Elo      | Rank |      Elo     | Rank |
| Vicuna  |  999 |    2 |     1011 (-47) |    1 |         1058 |    1 |
| Zephyr  | 1010 |    1 |      1003 (+3) |    2 |         1000 |    2 |
| GPT-3.5 |  991 |    3 |      993 (+52) |   3  |      941     |   3  |


> 2. **In Table 3, the example-correlation does not increase too much, comparing "All Reviewers (Weighted)" to "GPT-4 & Claude & GPT-3.5". These indicate that the core idea of PR may not be very useful.**
- "GPT-4 & Claude & GPT-3.5" is also an important variation of our PR approach of the weighted combination of all reviewers. The “All reviewers (Weighted)” includes two additional models. Both of them significantly outperform the baselines according to the significance test (>2%).

>3. **In Table 5, the performance boost with PD is not that great**
- We conducted a two-tailed t-test and results show that p-value of PD as compared to the baseline is smaller than 0.01, demonstrating significant improvement. – we will add this to the final version.

Reviewer comment (Presentation):
The two proposed methods are weakly related, except that they both introduce "peers" to some extent. The paper doesn't combine the two, which makes me feel cut off.

Author response:
> 4. **The two proposed methods are weakly related, except that they both introduce "peers" to some extent. The paper doesn't combine the two, which makes me feel cut off.**
- We politely argue that PR and PD are closely interrelated and fall under the same theme of providing a more fair (de-biased) ranking of long- and free-form answers. PR is for global ranking and PD is for fine-grained ranking.
- In Table 5 we compare their performance and provide insights on the difference between PR and PD and their benefits.
- Plus, we integrated them into one unified peer evaluation framework, i.e. first using PR to determine the weight of each LLM reviewer, and then conducting fine-grained PD among them, the first prunes out the weak LLMs and the second step enables more explainable evaluation decisions (discussion history). Through the study, we found that it achieves good efficiency and satisfiable results.
- We will add more details in the final version and adjust the corresponding part to make the connection sound more smooth.

Reviewer comment (Evaluation):
Introducing the communication between LLMs to a better solution will substantially increase computational cost. The paper does not seem to take this into consideration.

Author response:
> 5. **Introducing the communication between LLMs to obtain a better solution will require computational cost.**
- We admit that the cost of calling API-based commercial LLMs might be an issue, but it is more fair and is commonly used for evaluations in recent research literature. Such as for evaluating model performance on open-ended questions [3, 4],  summarization [5], dialogue generation [6], etc.
- Moreover, illustrating the decision process through multi-round LLM interactions is an essential part of recent work on model explainability, in topics such as evaluation [7], scalable oversight of model capabilities [8], and code optimization and text generation [9] – we will add this to the final version.
- In our work, we hired domain experts such as food science experts (from Upwork: [http://upwork.com](http://upwork.com)) for the human pairwise comparison annotations. For each pair comparison, the annotation takes on average 5-10 minutes. We pay \\$2-3 for each question. While in our experiments, GPT-4-based evaluation costs only about \\$0.015 for each question (based on token number). – Thus we believe using LLM for automatic evaluation is promising and cost-efficient.
- Beyond LLM-based automatic evaluation, another bigger scope theme of this work is about scalable oversight of models [10]. – Enable models to induce self-rankings that are hard for humans to compare.

Reviewer comment (Experiments):
For PR used in the win rate calculation, how will you determine whether it has converged? I only see the number of iterations in Algorithm 2 in Appendix E. Have you ever analyzed the impact of the number of iterations on the results?

Author response:
> 1. **For PR used in the win rate calculation, how will you determine whether it has converged? I only see the number of iterations in Algorithm 2 in Appendix E. Have you ever analyzed the impact of the number of iterations on the results?**
- We examine the impact of iteration quantity by manually contrasting outcomes for N iterations and significantly higher iterations, such as 2 * N or 3 * N. If the variations in each model's rating are less than 0.0001, we conclude that PR has reached convergence.

Reviewer comment (Reproducibility):
I need more information about the human annotation process. How exactly did the annotation process work? How did you recruit the participants and how about their backgrounds? How much were they paid?

Author response:
> 2. **I need more information about the human annotation process. How exactly did the annotation process work? How did you recruit the participants and how about their backgrounds? How much were they paid?**
- In our work, we hired domain experts such as food science experts (from Upwork) for the human pairwise comparison annotations. Before annotation, we provided a detailed instruction document to annotators and answered their questions. 
- For each pair of comparisons, annotators follow instructions on a UI shown in Appendix G, and the annotation takes an average of 5-10 minutes. We pay \$2-3 for each question.