Paper: JudgeLM : Fine-tuned Large Language Models are Scalable Judges (ICLR-2024)

Reviewer comment (Experiments):
The authors only compared the position bias between JudgeLM and GPT 3.5 or PandaLM, leaving the comparison on the two other biases to within the JudgeLM category. Moreover, they did not report these metrics on GPT-4, the teacher of JudgeLM, to examine if their proposed data augmentation strategies can also help JudgeLM supercede GPT-4 over these aspects.

Author response:
**Response 1:** We thank the reviewer for the question about comparison. We further analyze the three biases on PandaLM and apply the proposed methods to it, as shown in Table R8 and Table R9. We first transfer the 3.5K JudgeLM train samples into PandaLM format. Then, we train the PandaLM baseline, i.e., PandaLM*, with LLaMA-7B and the 3.5K train samples in PandaLM format. It can be observed that the PandaLM* also has position bias, which prefers the last answer. Besides, PandaLM* also faces knowledge bias and format bias, as shown in Table R9. Last, we apply the proposed methods to PandaLM*. As shown in Table R8 and Table R9, the proposed methods can also improve the performance of PandaLM*.

| Methods    | Agreement ↑ | Consistency ↑ | Bias ↓ (toward 1st) | Bias ↓ (toward 2nd) | Delta Bias ↓ |
|------------|:----------------------:|:---------------:|:-------------------:|:-------------------:|:--------------:|
| PandaLM*   |                 71.30 |         69.59 |              9.43 |             20.98 |       11.55 | 
| + swap aug. |                72.41 |          73.50 |              8.71 |             17.79 |        9.08 | 

Table R8. Ablation study of PandaLM for the swap augmentation on our val set.

| Methods  | ft w/ ref? | val w/ ref? | Agreement ↑ | Consistency ↑ | Bias ↓ (toward 1st) | Bias ↓ (toward 2nd) | Delta Bias ↓ |
|----------|:------------:|:-------------:|:----------------------:|:---------------:|:-------------------:|:-------------------:|:--------------:|
| *matching format.* |
| PandaLM* | ❎          | ❎           |                71.32 |         69.59 |              9.43 |             20.98 |       11.55 |
| PandaLM* | ✅          | ✅           |                75.15 |         74.08 |              8.62 |              17.30 |        8.68 |
| *mismatched format.*|
| PandaLM* | ❎          | ✅           |                68.67 |            65.00 |             15.82 |             19.18 |        3.36 |
| PandaLM* | ✅          | ❎           |                70.04 |         70.56 |              9.71 |             19.73 |       10.02 |
| *w/ ref. drop.* |
| PandaLM* | ref. drop   | ❎           |                71.88 |         71.56 |             11.24 |              17.20 |        5.96 |
| PandaLM* | ref. drop   | ✅           |                75.93 |         74.77 |              9.22 |             16.01 |        6.79 |

Table R9. Ablation study of PandaLM for the reference support and reference drop on our val set.

Reviewer comment (Experiments):
The authors argued that providing external knowledge to LLMs can alleviate the lack of related pre-trained knowledge. However, in their experiments, these "references" are essentially reference answers. Such information might be too sufficient for any LLM evaluation model, greatly decreasing the difficulty of the task, and is usually not available in most real-life scenarios (with only a few exceptions like judging homework submissions). Although incorporating reference answers does improve JudgeLM on validation data without reference, it could be better to conduct experiments where knowledge is injected under their retrieved form, instead of a well-organized answer.

Author response:
**Response 2:** We thank the reviewer for the suggested comparison with GPT-4. As shown in Table R10, we further list the metrics except for the agreement with GPT-4 itself. JudgeLM-33B achieves higher consistency than GPT-4, which demonstrates that JudgeLM-33B is more robust with position bias.

| Methods     | Agreement ↑ | Consistency ↑ | Bias ↓ (toward 1st) | Bias ↓ (toward 2nd) | Delta Bias ↓ |
|-------------|:-------------:|:---------------:|:-------------------:|:-------------------:|:--------------:|
| GPT-4       | -           |         85.82 |               6.10 |               8.10 |            2.00 |
| JudgeLM-33B |       89.03 |         91.36 |              5.55 |              3.09 |         2.46 |

Table R10. Comparison between GPT-4 teacher and JudgeLM-33B on JudgeLM val set.

Reviewer comment (Novelty):
The authors proposed three data augmentation strategies during fine-tuning. Yet among these methods, "reference drop" is basically equivalent to controlling the amount of samples with reference support, and thus should be a hyper-parameter of the latter strategy instead of an extra method.

Author response:
**Response 3:** Thanks for your great suggestion. As shown in Table R11, we further inject original reference answers into paragraphs with different words, and evaluate JudgeLM-33B with the injected paragraphs as references in a 0-shot setting. It shows that JudgeLM-33B can use references in the retrieved form with a certain loss of agreement and consistency.

| Reference Paragraph | Agreement ↑ | Consistency ↑ | Bias ↓ (toward 1st) | Bias ↓ (toward 2nd) | Delta Bias ↓ |
|---------------------|:-------------:|:---------------:|:-------------------:|:-------------------:|:--------------:|
| No                  |       89.32 |         92.37 |              3.62 |              4.01 |         0.39 |
| 50 words            |       87.78 |         92.35 |              3.00 |              4.65 |         1.65 |
| 100 words           |       87.69 |         91.84 |              2.62 |              5.54 |         2.92 |
| 200 words           |       86.77 |         90.48 |              2.70 |              6.82 |         4.12 |
| 300 words           |       86.25 |         89.26 |              3.13 |              7.61 |         4.48 |
| 400 words           |       85.59 |         89.00 |              2.98 |              8.02 |         5.04 |

Table R11. Performance of JudgeLM-33B with injected paragraphs as references on JudgeLM val set.

Reviewer comment (Evaluation):
In Table 3, is the total evaluation time of PandaLM computed based on 1 GPU only? If so, why do you not consider such a scenario where multiple (e.g. 8) PandaLMs run parallely on multiple machines with 1 GPU on each machine? This could be more fair to PandaLM, as all models use the same amount of GPU resources (which is the major computation resource).

Author response:
**Response 8:** We thank the reviewer for the advice. As mentioned in the **Efficiency Comparison** part of the global response, we made the ablations with only 1 GPU, and the JudgeLM is 16.65 times faster than PandaLM when it does not generate reasons.

Reviewer comment (Experiments):
Why is the data in Table 4 not aligned with the ones in Table 1? According to the paper, it seems like the final JudgeLM was fine-tuned on 100k data, which happened to appear in Table 4. Are there other factors that cause the difference, for example the number of training epochs?

Author response:
**Response 5:** Thanks for pointing that out. As shown in `6.2 SCALING ANALYSIS OF JUDGELM`, we analyze the scaling ability of the plain JudgeLM, which does not involve the proposed methods, i.e., swap aug, ref sup, or ref drop. We have underlined this in a revision of our paper.

Reviewer comment (Presentation):
In section 6.1, Table 1 is referenced as "Table. 1", which is different from other table references.

Author response:
**Response 6:** Thanks for pointing out the typo, we have fixed it in a revision of our paper.

Reviewer comment (Evaluation):
The use of GPT-4 generated judge for the fine-tuning data is the biggest weakness IMO, 1) using GPT-4 generated data could limit the usage of the proposed method (as the GPT-4 license), 2) the main evaluation of the judge system is the agreement with GPT-4, thus training on the GPT-4 generated judges may gives the proposed method a unfair advantage compared to other methods.

Author response:
Thanks for your important concern. We emphasize that the JudgeLM dataset is intended only for academic research and any commercial use is prohibited. Because the OpenAI's terms prohibit developing models that compete with OpenAI, the instruction-tuning datasets generated by the OpenAI's API, i.e., Alpaca, PandaLM, etc., all follow this rule. Thanks for your concerns, we have underlined this in a revision of our paper.

Reviewer comment (Experiments):
It could be nice if the paper could include some statistics of the type of question in the evaluation set? Like how many of them are on the topic of code generation or high school math?

Author response:
Thanks for your advice. We counted the distribution of questions in the JudgeLM val set as shown in Table R6, which has been added to a revision of our paper.

|            |  **culture** | **recommendation** | **finance** | **science** | **technique** | **common-sense** |   **art**   |    **math**   | **private-matter** |  **law**  |
|------------|:------------:|:------------------:|:-----------:|:-----------:|:-------------:|:----------------:|:-----------:|:-------------:|:------------------:|:---------:|
| count      |      233     |         482        |     142     |     393     |       42      |        373       |     335     |      250      |         421        |    204    |
| percentage |     4.66%    |        9.64%       |    2.84%    |    7.86%    |     0.84%     |       7.46%      |    6.70%    |     5.00%     |        8.42%       |   4.08%   |
|            | **planning** |    **roleplay**    |  **coding** |  **health** |  **writing**  |   **hardware**   | **history** | **geography** |     **others**     | **total** |
| count      |      309     |         77         |     201     |     278     |      625      |        130       |     243     |      199      |         63         |    5000   |
| percentage |     6.18%    |        1.54%       |    4.02%    |    5.56%    |     12.50%    |       2.60%      |    4.86%    |     3.98%     |        1.26%       |  100.00%  |

Table R6. Distribution of questions in JudgeLM val set.

Reviewer comment (Evaluation):
Since LLMs are currently still bad at certain reasoning tasks (such as counterfactual[R1]), how can we trust the evaluation results of such a judging system built by LLMs?

Author response:
Thanks for your concern. We agree with your concerns about LLM's reasoning performance at certain tasks. Nowadays, NLP researchers are still struggling with proposing LLMs with superior reasoning abilities. JudgeLM also needs the proposed `reference sup` method to enhance the judging ability for out-of-domain or counterfactual tasks, as shown in Fig. 11 and Fig. 16. Notably,  the proposed JudgeLM can benefit from stronger foundation LLMs, e.g., the LLaMA2-7B-Chat-based [1] JudgeLM outperforms the original JudgeLM-7B on all metrics, as shown in Table R7. We hold the same viewpoint as Reviewer rHPM, the research of judge models is critical for the development of LLMs and can benefit from advanced LLMs, establishing a positive cycle.

| Methods              | Agreement | Precision | Recall | F1-score | Consistency |
|----------------------|:--------:|:---------:|:------:|:--------:|:------------:|
|*w/o reference*|
| JudgeLM-7B w/ Vicuna       |   81.11  |   69.67   |  78.39 |   72.21  | 83.57 |
| JudgeLM-7B w/ LLaMA2-chat  |   83.87  |   73.43   |  80.06 |   75.91  | 85.17 |
|*w/ reference*|
| JudgeLM-7B w/ Vicuna       |   84.08  |   75.92   |  82.55 |   78.28  | 84.46 |
| JudgeLM-7B w/ LLaMA2-chat  |   86.60  |   79.47   |  83.11 |   81.02  | 87.74 |

Table R7. Comparison of different base models for JudgeLM-7B on JudgeLM val set.

[1] Llama 2: Open foundation and fine-tuned chat models

Reviewer comment (Evaluation):
One critical issue of the proposed method is its generalization ability to unseen tasks, which is important for an LM to be a general evaluation toolkit. A smaller LM, fine-tuned to distill the ability of a powerful proprietary LM, may experience a decrease in performance on unseen tasks.

Author response:
Thank the reviewer for the question. As mentioned in the **Generalization Ability** part of the global response, we apply JudgeLM to different benchmarks following the 0-shot setting and it achieves superior performance.

Reviewer comment (Evaluation):
The judge model lacks the granularity of its judgments and can only output an overall score. However, there may be many aspects to grade an open answer, e.g., factuality, fluency, novelty, and helpfulness. These fine-grained aspects are not considered in this work.

Author response:
Thank the reviewer for this suggestion. We further ask the JudgeLM to output the fine-grained results through the modified template. We also evaluate the JudgeLM-33B on the JudgeLM val set, as shown in Table R4.

|             | Accuracy | Precision | Recall | F1-score |
|-------------|:----------:|:-----------:|:--------:|:----------:|
| factuality  | 89.69    | 77.61     | 84.79  | 80.33    |
| fluency     | 88.35    | 79.86     | 82.35  | 81.01    |
| novelty     | 87.98    | 78.63     | 84.85  | 81.12    |
| helpfulness | 88.16    | 79.30     | 83.88  | 81.26    |
| all 4 aspects above     | 87.92    | 78.70     | 81.78  | 80.08|

Table R4. JudgeLM fine-grained evaluation results on JudgeLM val set.

Reviewer comment (Evaluation):
The writings related to efficiency are not very accurate. In “efficiency comparison”, the paper claims that JudgeLM’s superior efficiency is due to PandaLM “not support parallel running”. However, the parallel running is only a trivial implementation detail and should not be counted as a contribution. The main difference is that JudgeLM generates scores first followed by explanations and thus stopping generation before explanations can save time.

Author response:
Thanks for your question. As mentioned in the **Efficiency Comparison** part of the global response, we launched the ablations with only 1 GPU, and the JudgeLM still has superior efficiency when stops to generate reasons.

Reviewer comment (Experiments):
Is there any results of the performance in judging multiple answers? The paper has mentioned the format bias resulting from mismatched prompt format. However, the prompt template of judging multiple answers also mismatch that of judging two answers.

Author response:
We thank the reviewer for the advice. As shown in Table R5, we calculate the consistency results in the judging form of answer pairs and multiple answers. We first generate answers on JudgeLM val set through 3 LLMs, i.e., Vicuna-13B, LLaMA-7B, and alpaca-7B, for evaluation. Then we use pairwise judging and multiple judging to grade answers and rank them, respectively. Last, we compute the consistency between the two ranking results. Please note that `Error Rate@2` indicates the position orderings of two answers are different between the result of paired judgment and the result of multiple judgment, and `Error Rate@3` means the position orderings of three answers are different.

|             | Consistency | Error Rate@2 | Error Rate@3 |
|-------------|:----------:|:---------:|:---------:|
| JudgeLM-33B |   93.48 |   6.38 |   0.14 |

Table R5. Performance of JudgeLM-33B in judging multiple answers on JudgeLM val set. We calculate the consistency between the pairwise judging results and multiple judging ones.