An Empirical Study of LLM -as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT -4 (2025)

Abstract: Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have fine-tuned judge models based on open-source LLMs for evaluation. While the fine-tuned judge models are claimed to achieve comparable evaluation capability with GPT-4, in this work, we conduct an empirical study of LLM-as-a-Judge. Our findings indicate that although the fine-tuned judge models achieve high performance on in-domain test sets, even surpassing GPT-4, they underperform GPT-4 across several dimensions, including generalizability, fairness and adaptability. We also reveal that the fine-tuned judge model inherently operates as a task-specific classifier, consequently imposing the limitations.

## introduction recently, the evaluation for large-scale language models (llms) has drawn significant attention @xcite @xcite @xcite . some research has proposed llm-as-a-judge @xcite , namely utilizing proprietary llms, especially gpt-4 @xcite , to evaluate the llm's response. by defining evaluation schemes in the prompt template, proprietary llms can provide an accurate evaluation with high agreement with human evaluators. however, relying on external api for evaluation may introduce consideration about privacy leakage, and the opacity of api models also challenges the evaluation reproducibility. to address these issues, several fine-tuned judge models are proposed @xcite @xcite , relying on open-source foundation models and data constructed from either gpt-4 or human annotation, as shown in figure 1 . these models are validated on their respective meta-evaluation benchmarks, where the finetuned models exhibit performance on par with gpt-3.5 and gpt-4, leading to the affirmation of their evaluation capability. in this paper, we conduct an empirical study for the evaluation capability of judge models. experiment results indicate that while the fine-tuned judge models achieve superior accuracy on their respective in-domain test sets, they still exhibit limitations compared with close-sourced proprietary models: we argue that these limitations primarily stem from the fine-tuning process, where the foundation model is transformed into a task-specific classifier overfitted to the fine-tuning data. to draw a conclusion, the fine-tuned judge model cannot serve as a general substitute for gpt-4 in terms of llm evaluation. it is advisable to exercise caution when leveraging them for evaluation in real applications, watching for the overlap between the evaluation scenario and the fine-tuning process. in this section, we make a comprehensive empirical study based on four representative fine-tuned judge models in ## the essence of fine-tuned judge: a task-specific classifier combining all the limitations revealed in our experiments, we would like to claim that after the finetuning process on a single task, the judge model has degenerated into a task-specific classifier, which is overfitted to the training data. as shown in we also analyze the correlation between different predictions made by different evaluators. as shown in figure 2 and 3, the correlation among different classification models is much closer than their correlation with gpt-4. different as they are in architectures, all three models are inherently classifiers fitting to the same set of supervision, leading to similar evaluation outcomes. although prior research on instruction-tuning all emphasizes the importance of data diversity @xcite , the fine-tuning of judges is doing the opposite thing. therefore, after fine-tuning for a single task with a fixed prompt template, the model lost its generalization ability, and degenerate into a task-specific classifier, which exhibits several limitations due to overfitting. ## conclusion although the fine-tuned models demonstrate superior performance on in-domain test sets, they still have several limitations compared to gpt-4. while increasing the fine-tuning data could possibly mitigate some of the limitations, as the potential of llm extends beyond boundaries, there will always be new domains and tasks that are not covered by the fine-tuning scope. therefore, the fine-tuned judge model cannot replace gpt-4 as a universal evaluator for llms, and should be used judiciously by watching the domain and task adaptability.