Do Large Language Models understand how to be judges? (2025)

Abstract: This paper investigates whether Large Language Models (LLMs) can effectively act as judges for evaluating open-ended text generation tasks, such as summarization, by interpreting nuanced editorial criteria. Traditional metrics like ROUGE and BLEU rely on surface-level overlap, while human evaluations remain costly and inconsistent. To address this, we propose a structured rubric with five dimensions: coherence, consistency, fluency, relevance, and ordering, each defined with explicit sub-criteria to guide LLMs in assessing semantic fidelity and structural quality. Using a purpose-built dataset of Italian news summaries generated by GPT-4o, each tailored to isolate specific criteria, we evaluate LLMs’ ability to assign scores and rationales aligned with expert human judgments. Results show moderate alignment (Spearman’s ρ = 0.6–0.7) for criteria like relevance but reveal systematic biases, such as overestimating fluency and coherence, likely due to training data biases. We identify challenges in rubric interpretation, particularly for hierarchical or abstract criteria, and highlight limitations in cross-genre generalization. The study underscores the potential of LLMs as scalable evaluators but emphasizes the need for fine-tuning, diverse benchmarks, and refined rubrics to mitigate biases and enhance reliability. Future directions include expanding to multilingual and multi-genre contexts and exploring task-specific instruction tuning to improve alignment with human editorial standards.

## introduction evaluating open-ended text generation depends on a set of often implicit criteria that are hard to formalise. traditional metrics like rouge @xcite , bleu @xcite , and meteor @xcite reduce evaluation to surface-level overlap, overlooking deeper qualities such as semantic fidelity and target-audience relevance @xcite . human judgments capture these nuances but are costly, inconsistent, and difficult to scale @xcite . large language models (llms) offer a potential solution: given a clear, multi-item criterion, they could score each criterion and supply a rationale, promising consistency and low cost. yet this hinges on whether llms actually understand the criterion's language and hierarchy. for example, when asked to evaluate summaries, can an llm reliably distinguish between objective bullets (e.g., does this summary include every key claim?) and subjective ones (e.g., is the tone appropriate?), and combine them into a coherent overall score? in this paper, we test several llms using few-shot ## related works the use of large language models (llms) as evaluative judges has emerged as a prominent methodology for assessing ai-generated outputs. these systems can be broadly classified into three categories: prompted judges, fine-tuned judges, and multi-agent judges. prompted judges rely on the intrinsic capabilities of llms, activated through carefully engineered prompts, without requiring additional training @xcite @xcite . fine-tuned judges, in contrast, are explicitly trained on specialized preference datasets to enhance their evaluation precision @xcite @xcite @xcite . these models are often fine-tuned using data sourced from human annotations or distilled judgments from advanced models like gpt-4 @xcite . despite their robust performance on benchmarks, fine-tuned judges frequently fail to generalize effectively across diverse or unfamiliar tasks, as noted by @xcite . this limitation arises partly because the datasets used for fine-tuning typically lack sufficiently complex examples, thereby constraining the reasoning capabilities of these judges. finally, multi-agent judges employ a collaborative approach, leveraging the outputs of multiple llms in a sequential or ensemble framework to generate judgments @xcite @xcite . although this approach offers enhanced evaluation robustness by surpassing the abilities of a single model, it incurs significantly higher computational costs during inference. as llm-based judges gain widespread adoption for evaluating and refining large language models, numerous benchmarks have been developed to assess their performance. prominent examples include llmeval @xcite , mtbench @xcite , and faireval @xcite , which emphasize alignment between llmbased judges' assessments and human evaluations. however, these benchmarks are often constrained by the subjectivity inherent in human evaluation, which can prioritize stylistic elements over factual and logical accuracy. in response, llmbar @xcite introduces a methodology that evaluates judges' ability to adhere to instructions, employing response pairs with clear ground-truth preference labels. conversely, judgebench focuses on more complex tasks, such as evaluating reasoning capabilities and distinguishing between correct and incorrect responses, surpassing the scope of simple instructionfollowing tasks. it generates challenging response pairs for evaluating llm-based judges using a robust pipeline to transform datasets with ground truth labels into pairs where one response is correct and the other is not. the pipeline ensures stylistic consistency and mitigates biases like self-enhancement. it filters out questions where all responses are entirely correct or incorrect, making it harder for llm judges to distinguish. judgebench can adapt diverse datasets, including knowledge, reasoning, mathematics, and coding. for reward models, rewardbench @xcite provides a comprehensive evaluation across domains such as safety, dialogue, and reasoning. this benchmark aggregates multiple preference datasets and prior benchmarks @xcite @xcite @xcite @xcite @xcite @xcite , enabling a holistic assessment of reward models' performance. ## method to investigate whether llms can "understand" and apply given evaluation criteria, we designed an experimental setup that centers on how well an llm internalises and operationalises each rubric item. rather than relying on off-the-shelf summarization benchmarks (whose reference summaries are often misaligned with editorial standards, prone to test contamination, and insufficiently detailed), we constructed a purpose-built corpus and rubric explicitly tailored for probing criterion interpretation. ## experiments we evaluated twelve slms without applying any fine-tuning or softtuning, relying solely on a few-shot prompting approach to ensure a fair comparison. each model was prompted using a standardized template (appendix a) designed to guide assessments based on five established editorial criteria. for each criterion, the prompt instructed the model to act as an impartial evaluator, assigning a score from 1 to 5 and providing a detailed explanation in italian that justifies the rating. the prompt includes the definition of the criteria and detailed descriptions for each score level to standardize expectations. each criterion was further defined through a set of sub-criteria that specify key aspects for evaluation. the expected output followed a predefined json format, requiring both the numerical score and a rationale. summeval @xcite , a meta-evaluation dataset for summarization, was used to construct 15 few-shot examples for coherence, consistency, fluency, and relevance to guide model predictions. for the ordering criterion, we generated synthetic summaries using gpt-4o to illustrate varying levels of quality. we also generate explanations that are manually reviewed to ensure alignment with the editorial criteria. this setup enabled an intrinsic evaluation of the models' ability to assess summarization quality independently of external training data. for comparison, we also evaluated selected llms under the same conditions to establish an upper performance bound. ## results the analysis of model performance across various families, illustrated in figures 1 , 2 , and 3, reveals that scaling effects are not uniform and depend on both the model family and the specific metric considered. for example, the deepseek models (red markers in figure 1 ) demonstrate a clear reduction in mean absolute error (mae) as model size increases, as depicted in figure 2 . the 1.5b parameter model exhibits an mae of 1.47, which improves to 1.05 for the 14b variant. however, figure 3 shows that the corresponding spearman's ρ values for deepseek models fluctuate near zero (ranging from -0.018 to 0.031) across these sizes. this divergence is also evident in figure 1 , where deepseek models cluster on a line around a spearman's ρ of zero while mae decreases. this suggests that while increased parameters can improve absolute error metrics, they do not necessarily enhance the model's ability to rank predictions in alignment with the evaluation target for this family. in the gemma 3 family (blue markers in figure 1 ), figure 2 indicates a u-shaped trend for mae with increasing model size; the 1b model has an mae of 1.84, the 4b model records a lower mae of 1.38, and the 12b model shows an mae of 1.53. concurrently, figure 3 highlights that the 4b model yields a statistically significant negative spearman's ρ of -0.179 (@xmath0.005), while the 1b and 14b models yield ρ values closer to zero (-0.078 and 0.010, respectively). these findings, also visible in figure 1 , where the 4b gemma model stands out with its negative correlation, indicate that only specific scales within this family show notable differences in ranking performance, raising questions about non-linear scaling effects. for llama 3 models (black markers in figure 1 ), the trend is less clear regarding mae improvement with scaling. figure 2 shows that the 2b, 3b, and 8b variants produce similar and relatively stable mae values (ranging from 1.34 to 1.51). correspondingly, figure 3 demonstrates that spearman's ρ values for llama models are consistently near zero across these sizes (varying between -0.005 and 0.027). figure 1 further confirms this, with llama models tightly clustered around zero correlation. this general lack of significant change in either mae or correlation across different sizes suggests that scaling within the tested llama family range may not substantially impact either absolute accuracy or rank consistency for this task. in the phi family (yellow markers in figure 1 ), figure 2 shows that the 3.8b model achieves a mae of 1.44, which slightly improves to 1.31 for the 14b model. however, figure 3 reveals a striking contrast in correlation: the 3.8b model has a moderate positive spearman's ρ (ρ = 0.219, @xmath1.001), while the 14b model's ρ drops to a negligible value (0.010, @xmath2.874). this pattern, clearly distinguishable in figure 1 , implies that reducing absolute error does not guarantee enhanced ordinal ranking of predictions and can even correspond to a decrease in ranking performance for this family. the qwen models (purple markers in figure 1 ) exhibit more complex scaling dynamics. as seen in figure 2 , mae for qwen models does not follow a simple trend: the 0.5b model has an mae of 1.36, which dips for the 1.8b model (mae 1.24), then rises for the 4b (mae 1.54) and 7b (mae 1.71) models, before slightly decreasing for the 14b model (mae 1.46). spearman's ρ, shown in figure 3 , also fluctuates: the 0.5b model has a ρ of 0.137 (@xmath3.031), which then varies for larger models (1.8b to 14b) between approximately 0.040 and 0.120. this variability, also reflected in the scatter of qwen points in figure 1 , indicates that scaling within the qwen family has a somewhat unpredictable impact on both absolute error and ranking performance. notably, the gpt family (green markers in figure 1 ) reveals that "mini" variants can achieve both low mae and comparatively stronger rank correlations. for instance, gpt-4o-mini shows an mae of 1.04 and a spearman's ρ of 0.277 (@xmath4.000), the highest correlation observed among the gpt models plotted. similarly, gpt-4-turbo-mini (equivalent to text's gpt,4.1 mini) records an mae of 1.42 with ρ = 0.173 (@xmath5.006). as seen in figure 1 , these contrast with other larger gpt versions, such as gpt-4o (mae 1.15, ρ = 0.018) and gpt-4 (mae 1.42, ρ = 0.192), where the correlation, while sometimes positive, can be less pronounced than the topperforming mini variant. this suggests that a reduced architecture in this family might, in some cases, better capture the ordering of predictions. in summary, the results indicate that while scaling can sometimes reduce absolute prediction error (mae), it does not systematically improve the preservation of ordinal relationships as measured by spearman's ρ. the diverse trends across model families support the view that model improvements should be evaluated on a case-bycase basis, considering both error minimization and rank correlation. future work should investigate the architectural and training factors that contribute to these complex dynamics, with particular attention to why some models or families (such as certain gpt mini variants or the smaller phi model) achieve better ranking performance relative to their size or absolute error. positive bias the bar plots shown in figures 4 and 5 comparing human and model ratings across the five editorial criteria reveal a consistent pattern of positive bias in model-generated evaluations. most language models tend to assign higher scores than human annotators, particularly in subjective dimensions such as fluency and coherence. this trend is observed across multiple model families and parameter scales, suggesting a global rather than local phenomenon. this bias cannot be attributed to imbalanced prompting. the few-shot examples used to guide model behaviour were carefully constructed to span the full range of the scoring scale (1-5), ensuring that models were exposed to both high and low quality examples in equal quantities. this design choice rules out the possibility that models are simply mimicking overly generous examples. a more plausible explanation lies in the interplay of training data biases and alignment methodologies. many evaluated models are pre-trained on large-scale synthetic corpora, often generated by other language models or curated to reflect "high-quality" outputs, which may encode implicit biases toward agreeableness or flattery @xcite . this aligns with findings that sycophantic tendencies can emerge from overfitting to human preferences during reinforcement learning from human feedback, where annotators disproportionately favor responses that align with their views @xcite . for instance, studies show that even pretrained models exhibit sycophancy, likely due to absorbing patterns from internet texts where users often reinforce shared opinions (e.g., reddit discussions) @xcite . additionally, the observed bias may stem from the models' tendency to "flip-flop" when challenged, altering answers to align with user suggestions even when initially correct. this behavior, quantified in experiments like flipflop @xcite , reveals that state-of-the-art llms (e.g., gpt-4, gemini-pro) frequently compromise accuracy to maintain user agreement, with sycophantic responses occurring in over 58% of cases @xcite . such dynamics are exacerbated by alignment objectives prioritizing politeness and helpfulness over factual rigor, inadvertently discouraging critical pushback @xcite . notably, while finetuning on synthetic datasets, balancing confirmatory and corrective responses can reduce sycophancy by 50% in some models (e.g., mistral-7b), the persistence of regressive sycophancy (where agreement leads to incorrect answers) underscores the need for robust mitigation strategies that reconcile alignment with truthfulness @xcite . the individual bar plots comparing human and model ratings across the five editorial criteria reveal a consistent pattern of positive bias in model evaluations. across nearly all models and criteria, the distributions of model-assigned scores are skewed toward higher values relative to human annotations. this trend is particularly pronounced in subjective dimensions such as fluency and coherence, where models frequently assign ratings of 4 or 5, even in cases where human annotators opt for more moderate scores. for instance, in the fluency barplots shown in figure 6 , models like gpt-4.1 mini, gemma 3 1b, and qwen 3 8b con-sistently overrepresent high scores, suggesting a systematic overestimation of linguistic quality. similarly, in coherence barplots 6, models often rate summaries more favorably than human experts, with fewer low scores and a concentration around the upper end of the scale. this positive skew is not limited to a specific model family or size. it appears across both small and large models, including reasoning variants. this suggests that the bias is not merely a function of model capacity but may reflect shared training dynamics or evaluation heuristics. in conclusion, while llms show promise as scalable evaluators, their tendency to overrate outputs highlights the need for calibration. future work should explore methods to mitigate this bias, such as incorporating human-aligned calibration datasets, adversarial prompting, or ensemble evaluation strategies that combine model and human judgments. ## conclusions three themes are emerging from our experiments on how off-the-shelf llms behave when asked to judge outputs against a fixed rubric. first, size matters, but only up to a point. as models grow larger, they generally make fewer absolute errors in scoring, which might lead you to think "bigger is always better". yet when we look at how well these scores line up in rank order with human judgments, the picture is more mixed. some of the smaller "mini" variants do a better job of getting the ordering right than their much larger siblings. in other words, raw scale helps with scoring precision but doesn't automatically translate into human-like ranking ability. second, almost every model we tested leans on the generous side. they tend to hand out higher scores than human experts do, especially on subjective dimensions like fluency or coherence. this consistent positive bias suggests that the models' pretraining and alignment processes prime them to sound "helpful", perhaps at the expense of rigour. in practice, it means you can't assume their high marks carry the same weight as an expert's. ultimately, you'll see that agreement between models follows a similar pattern: small or architecturally distinct models often disagree wildly, whereas larger models within the same family converge on very similar judgments. so if you're looking for consistency between multiple llm-based judges, you'll get it only once you reach a certain size threshold. putting all of this together, we conclude that llms are capable of approximating human scores, but they still struggle with unbiased ranking and inter-model consensus at smaller scales. this could stem from a lack of un understanding of the evaluation rubric. moving forward, targeted calibration techniques and a closer look at what makes some "mini" models better rankers might hold the key to get more reliable automated judges that better understand the scoring criteria.