Rating Roulette: Self-Inconsistency in LLM -As-A-Judge Frameworks (2025)

Abstract: As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.

## introduction as natural language generation (nlg) becomes more prevalent in a wide variety of applications like automated journalism, customer service chatbots, language translation and content summarization, proper evaluation and measurement of alignment with human preference are critical to improve user trust and system utility. although traditionally used automatic metrics like bleu @xcite and bertscore @xcite work well with tasks like translation when multiple references are available, they fail to work in more open-ended tasks like summarization or general chatbot settings. to automatically evaluate llms in these settings, llm-as-a-judge @xcite has emerged as an automatic, scalable alternative to manual evaluation. the common practice is to prompt an llm to evaluate natural language generations. the ratings obtained are then verified by comparing with ratings assigned by human judges, which are taken as the gold standard. this comparison is commonly done through computing exact match for categorical labels or correlation for numeric or ordinal scales @xcite . however, what is often missing from these studies is any measure of self-reliability or intra-rater reliability of both llm and human judges. we define self-reliability as the agreement of a judge with itself over multiple runs with the same settings (prompt and hyperparameters in case of llms). meanwhile, self-reliability data is completely missing from human annotations in existing benchmarks. in some cases, annotations from multiple human judges exist, but they only give us information about the inter-rater reliability instead, which is also frequently lower than the conventionally agreed upon agreement thresholds @xcite @xcite . meanwhile, agreement between llm and human judges is usually computed using metrics like correlation and accuracy, instead of using metrics specifically designed for measuring agreement like krippendorff's alpha. this can often lead to an overestimation of agreement since those metrics do not account for chance agreement @xcite . nlg can be evaluated using different metrics on a variety of tasks such as machine translation, dialog generation and summarization. in our experiments, we first study the simplest case of evaluating summarization -assigning a binary label to a summary in the summac benchmark @xcite . next, we look at evaluating summaries in more complex scenarios, using a likert rating scale of 1 to 5 to rate a summary on several metrics like coherence, consistency, fluency and relevance. we can use the summeval benchmark for this purpose. finally, we look at evaluating a different task of nlg, where we use a judge llm to rank multi-turn conversations from two competing llms in the mt-bench benchmark @xcite . in this work, we make several key contributions to understanding the reliability of llm-generated ratings in nlg evaluation. first, show that ratings output by llms have low agreement over multiple runs with the same prompt. second, we show that turning off any sampling to make an llm always output the same rating hurts performance measured by agreement with human judgment. third, we find that this phenomenon persists across multiple tasks and benchmarks related to nlg, indicating that this is a widespread problem that needs to be addressed. finally, we discuss some recommendations to conduct more robust nlg evaluations. llm-as-a-judge large language models can serve as judges by assigning scores, rankings, or labels to generated outputs, a paradigm surveyed in recent works @xcite . llm judges have been applied across 20 nlp tasks @xcite and integrated into ai-assisted human evaluation @xcite . notably, gpt-4 with chainof-thought prompts aligns more closely with human judgments than conventional automatic metrics on nlg tasks @xcite @xcite . evaluation typically involves comparing llm outputs to human judgments via correlation @xcite @xcite or percentage agreement @xcite on benchmarks such as judgebench @xcite . challenges include bias @xcite , uncertainity in judgments @xcite , adversarial vulnerability @xcite , and domainspecific performance gaps @xcite , in addition to existing problems of llm generations like prompt sensitivity @xcite . to enhance reliability, methods like panel-based evaluation (poll) @xcite and comparative studies of fine-tuning versus gpt-4 prompting @xcite ) have been proposed. there is another challenge, high variability in llm outputs leading to inconsistencies in llm judges. variability in llm judgments due to the stochastic nature of llms, they give different outputs when run on the same prompt multiple times. while this is by design, it should not alter the ratings assigned by an llm judge to the same rating. @xcite observed some variability in ratings produced by llm judges on the writing-prompts @xcite dataset. @xcite took advantage of this variability in gpt-4 by setting temperature to 1 and sampling 20 times to get multiple scores and then getting the final rating by normalizing all the scores by their probabilities. in our work, we go further to analyze the implications of high variability in assigned judgments, discuss metrics to measure this variability, study the extent of this issue across different tasks and benchmarks, and explore whether we would get better results by turning off the variability by setting the temperature to zero. we frame this variability as self-reliability or intra-rater reliability. self-reliability also called intra-rater reliability, this measures the consistency of a single evaluator's judgments across repeated assessments, as opposed to inter-rater reliability. common metrics include the intraclass correlation coefficient (icc) for continuous scales @xcite ), cohen's kappa for categorical ratings, and krippendorff's alpha for ordinal, interval, or ratio data with missing values @xcite . reporting intra-rater reliability is standard in domains like essay grading @xcite , physical therapy @xcite , and clinical diagnostics @xcite , and its joint consideration with inter-rater reliability is urged by @xcite . yet, nlp annotation and llm-evaluation studies frequently omit selfreliability metrics @xcite , a deficiency we confirm in our llm-as-a-judge experiments, where models exhibit unstable selfjudgments. ## experiments we first consider a simple framework for nlg evaluation, binary classification of machine-generated summaries. a good candidate for this is summac, a factual consistency detection benchmark. following this, we make a deeper dive into the performance of llm judges on the summeval benchmark @xcite . compared to the binary labels in the summac benchmark, the summaries in this benchmark are evaluated by multiple human raters on a likert scale across multiple metrics. @xcite introduced the mt-bench benchmark, which evaluates an llm's multi-turn conversational and instruction-following ability. while related to nlg, it is a very different task compared to summarization. ## intra-rater reliability results summac summeval in figure 4 , we see that compared to human evaluators, the different runs in deepseek-r1 and qwen 3 show a high self-reliability on coherence and consistency, with very low selfreliability on fluency. not only does fluency have low reliability for all our llm judges, but it is also the only metric where llama performs best. this suggests that an llm judge's ability to reliably produce ratings depends heavily on the metric. ## conclusion we present a comprehensive analysis of intra-rater reliability in llm-as-a-judge frameworks, revealing key challenges for evaluation in this domain. our experiments show that llms display low selfreliability when evaluating the same content across multiple runs, even with identical prompts and hyperparameters. this inconsistency persists across various tasks and metrics. although newer, larger models like qwen 3 are more consistent than models such as llama 3.1, they still often fall short of standard reliability thresholds. our work carries some potential risks. we have measured the performance of llm judges by their agreement with human judges, which can be noisy depending on the inter-annotator agreement of the human labels. llm judgments could also potentially have biases for certain linguistic styles which could influence the scores presented in this paper. despite these risks, our findings have important implications for llm-as-a-judge research. reporting single-run llm judgments without consistency metrics can be misleading, and certain aspects of text quality, such as fluency, remain difficult to assess reliably for both humans and machines. also, the pronounced disagreement between expert and crowdsourced judges highlights the need to clarify which human preferences llms should model. while llm-as-a-judge approaches offer scalability, they face significant reliability challenges. addressing these issues is essential for developing evaluation frameworks that meaningfully capture distinctions in text quality.