Judging the Judges: Evaluating Alignment and Vulnerabilities in LLM s-as-Judges (2025)

Abstract: The LLM-as-a-judge paradigm offers a potential solution to scalability issues in human evaluation of large language models (LLMs), but there are still many open questions about its strengths, weaknesses, and potential biases. This study investigates thirteen models, ranging in size and family, as ‘judge models’ evaluating answers from nine base and instruction-tuned ‘exam-taker models’. We find that only the best (and largest) models show reasonable alignment with humans, though they still differ with up to 5 points from human-assigned scores. Our research highlights the need for alignment metrics beyond percent agreement, as judges with high agreement can still assign vastly different scores. We also find that smaller models and the lexical metric contains can provide a reasonable signal in ranking the exam-taker models. Further error analysis reveals vulnerabilities in judge models, such as sensitivity to prompt complexity and a bias toward leniency. Our findings show that even the best judge models differ from humans in this fairly sterile setup, indicating that caution is warranted when applying judge models in more complex scenarios.

## introduction over the last few years, large language models (llms) have demonstrated remarkable capabilities across various domains @xcite @xcite . as more and more new llms with different architectures and training methods continue to be released and their capabilities expand, accurately evaluating their performance and limitations becomes increasingly challenging @xcite @xcite @xcite . llm evaluation methods generally fall into one of two broad categories. benchmarks such as * equal contribution mmlu @xcite , truthfulqa @xcite , and gsm8k @xcite assess specific capabilities, while leaderboards such as chatbot arena @xcite and open llm @xcite rank models based on human or automated pairwise comparisons. both approaches face challenges in evaluating free-form text responses, as assessment can be as difficult as generation itself (see e.g. @xcite . one approach to evaluating llms is using mcq benchmarks like mmlu, which compare answer log-probabilities instead of assessing generated responses directly. however, this approach limits the range of measurable abilities and differs from how llms are used in practice. lexical methods, such as exact match (em) or n-gram overlap, are practical and cost-effective but prone to false negatives and often miss subtle semantic differences. these challenges are amplified for instruction-tuned chat models, which tend to produce more verbose responses @xcite . for these reasons, human evaluation remains the gold standard for evaluating llm responses. human evaluation is, however, expensive and often impractical, leading to the growing use of llms as judge models @xcite @xcite . while promising alignment with humans has been noted @xcite , questions about this approach remain. this work examines llms as judges, contrasting them with humans and automated methods. unlike prior studies, we focus on scenarios with high human alignment to separate task ambiguity from judge model limitations. using triviaqa @xcite , we evaluate how judge models of varying architectures and sizes assess exam-taker models. in this work, we study the properties of llms as judges, comparing them with humans and auto-l l a ma 2 -7 b ## related work various recent studies have used or considered using llms as judges for tasks such as evaluating story generation (chiang and lee, 2023), retrieval-augmented generation @xcite , visual qa @xcite , code comprehension @xcite , multilingual evaluation @xcite and more general open-ended tasks @xcite . @xcite and @xcite propose ways to standardise llm evaluations and the role that judge models might play in such solutions. several studies have demonstrated that state-of-the-art llms such as gpt-4 turbo exhibit high alignment with human judgments @xcite , though others also illustrate that the paradigm is not yet without faults. @xcite propose a benchmark for evaluating the performance of llms as judges, and other approaches have been proposed to improve llm judges such that they are aligned well with humans @xcite . despite promising results in various settings, judge models still suffer from known issues of current llms such as hallucinations and factual errors @xcite and difficulty in following complex instructions @xcite . furthermore, various studies have reported challenges such as position bias @xcite @xcite , verbosity bias @xcite in their preferences, confusing evaluation criteria @xcite , or focusing more on the style and grammar compared to factuality @xcite . recently, @xcite have shown that llms perform better in comparative assessment compared to absolute scoring, which can be used for reliably measuring the relative per-formance of models @xcite and creating classifiers for pairwise grading @xcite . we build on previous work to investigate the strengths and weaknesses of llms as judges. unlike previous studies, we focus on comparing llm outputs with reference answers rather than pairwise comparisons on open-ended tasks. with high human alignment in this setting, we gain a clearer view of llm performance. furthermore, we extend previous research by considering more llms, both as judges and as evaluated models. ## methodology to evaluate the strengths and weaknesses of the llm-as-a-judge paradigm, we focus on a comparatively controlled setup, in which judge models assess answers of exam-taker models on the knowledge benchmark triviaqa @xcite . with this methodological design, it is possible to focus on the abilities of the judges in isolation, without having to address human disagreement and error at the same time. in this section, we elaborate the main aspects of our methodology. evaluation data as our testbed, we use the triv-iaqa dataset @xcite , consisting of 95k question-answer pairs sourced from 14 trivia and quiz league websites. each question in the train and validation set is annotated with a list of short answers containing a minimal set of facts and evidence documents collected from wikipedia and the web. for our experiments, we use the validation set of the unfiltered partition of the benchmark, using the short answers as reference answers. we use the training set for few-shot examples. since experiments require manual annotation of the exam-taker model responses, we use a random sample of 400 questions from the dataset. in appendix i, we show with a bootstrapping test that this sample size has low variance for our main result. through experiments described in § 3, we establish that humans have high agreement on judgements of answers given to the questions in the benchmark. exam-taker models to understand the strengths and weaknesses of different judges, we consider answers of pre-trained (base) and instruction-tuned (chat) 'exam-taker models' across a wide variety of model sizes. in particular, we consider llama-2 @xcite in 7b, 13b, and 70b parameter sizes for both base and chat versions, mistral 7b (jiang et al., 2023) base and chat versions, and gpt-4 turbo foot_0 (achiam et al., 2023) as the examtaker models. the prompts for the exam-taker models contain five few-shot examples of (question, answer) pairs from the triviaqa training set. the prompts for the instruction-tuned models additionally include a command signaling the model to answer the given question in a succinct manner similar to the provided examples. the prompts are provided in appendix d. judge models to get a comprehensive view of the strengths and weaknesses of judge models across different model sizes and architectures, we use instruction-tuned versions of llama-2 (touvron et al., 2023) in 7b, 13b, and 70b sizes, llama-3 (ai@meta, 2024) in 8b and 70b sizes, llama-3.1 (dubey et al., 2024) in 8b and 70b sizes, mistral 7b (jiang et al., 2023), gpt-4 turbo (achiam et al., 2023), gemma 2b @xcite , and judgelm 7b @xcite as judges. to maintain parity with human and judge evaluation, judge prompts were built from human guidelines in appendix g. the judges are instructed to respond with only a single word, "correct" or "incorrect". an overview of all exam-taker models and judge models is shown in baselines as baselines, we use two commonly used lexical evaluation techniques -exact match (em) and contains match (contains). for em, a response is considered correct if the response exactly matches one of the reference answers for the given question. for contains, an answer is considered correct if at least one of the reference answers is a sub-string of the response string. both em and contains match are computed in a case-insensitive manner. alignment we use two metrics to quantify alignment between judges: percent agreement and scott's pi coefficient @xcite ). foot_2 percent agreement expresses a simple percentage of the samples on which two annotators agree. scott's pi, denoted as scott's π, is an alignment metric that corrects for chance agreement between two annotators and is considered to provide a more robust measure of alignment. details about the computation of both metrics are given in appendix f. human judgements as a ground-truth assessment, we obtain human annotations for each examtaker model answer. the inter-human alignment is calculated between three human judges using the answers to 1200 randomly sampled questions answers; the human guidelines can be found in appendix g. we then determine collective "human judgment" through a majority vote. the average alignment between human evaluators and the majority vote yielded a scott's π of 96.2 ± 1.07, foot_3 while the average percentage agreement was 98.52% ± 0.42%, exceeding the alignment previously reported in comparable studies @xcite . the details of this experiment are mentioned in appendix a. given this near-perfect alignment score, we consider only one human evaluator per sample for the rest of our experiments, to reduce the overall cost of human annotations. the set of questions for which we obtain human annotations is identical for each exam-taker model. ## results in this section we discuss our main results, primarily focusing on the relationship between evaluations by various judge models and human evaluations ( § 4.1), and how that impacts their usability ( § 4.2). to do so, we evaluate their alignment with human judgment and assess how differently they rank the nine exam-taker models compared to humans. in section 5, we further analyse their precision and recall to further investigate the types of errors that can be made by various judge models. details about compute requirements and others costs for experiments are given in appendix h. ## analysis to better understand the judge models, we conduct multiple case studies aimed at identifying common errors and vulnerabilities in the judges we investigate. specifically, we study their precision and recall and error types ( § 5.1), their sensitivity to the instruction prompt prompt ( § 5.2), how they respond to controlled resposes of specific types ( § 5.3), and the extent to which they have a leniency bias ( § 5.4). ## conclusion in this work, we conduct an extensive study of llms as judges, comparing them to human judges and automated evaluation methods. by focusing on a clean evaluation scenario with high inter-human agreement, we identify potential issues with the llm-as-a-judge paradigm, separate from task ambiguity. we find that smaller, cost-efficient models, like mistral;7b, are less effective than larger models such as gpt-4 turbo, llama-3.1;70b, and llama-3;70b, which are better aligned but still fall short of human alignment. even with high alignment, their scores can differ by up to 5 points from human scores, highlighting the need for caution when using judges in more complex scenarios. we also note that the commonly used metric of percent aligned fails to differentiate between judges effectively. we suggest future work adopt the more robust scott's π metric for better distinction. next, we note that high alignment scores are not always necessary to discriminate between models. while gpt-4 turbo and llama-3 have excellent alignment scores, simpler and more cost-efficient models, like contains, perform similarly in ranking exam-taker models, despite lower alignment scores and score deviations. for studies focused on ranking models rather than estimating exact scores, these approaches can be as suitable as more expensive ones. lastly, we run experiments to assess judge models' sensitivity to prompts, precision, recall, error types, leniency, and vulnerability to dummy answers. we find that smaller models are more likely to judge positively when in doubt, that loweralignment models lack precision, and that better models are more robust across different prompts but harder to "steer." some judge models are easily fooled by dummy answers like "yes" and "sure" and are better at detecting completely incorrect answers than partially incorrect ones. overall, this work contributes to llm evaluation by assessing judges in a clearly defined framework. our results highlight the potential of llms as judges but caution against blindly trusting their judgments, even when aligned with humans. we recommend computing both percent agreement and scott's π, paired with qualitative analysis, to avoid bias. we discuss limitations in appendix a and plan to expand our work to more complex scenarios in the future.