From Calculation to Adjudication: Examining LLM Judges on Mathematical Reasoning Tasks (2025)

Abstract: To reduce the need for human annotations, large language models (LLMs) have been proposed as judges of the quality of other candidate models. The performance of LLM judges is typically evaluated by measuring the correlation with human judgments on generative tasks such as summarization or machine translation. In contrast, we study LLM judges on mathematical reasoning tasks. These tasks require multi-step reasoning, and the correctness of their solutions is verifiable, enabling a more objective evaluation. We perform a detailed performance analysis and find that easy samples are easy to judge, and difficult samples are difficult to judge. Our analysis uncovers a strong correlation between judgment performance and the candidate model task performance, indicating that judges tend to favor higher-quality models even if their answer is incorrect. As a consequence, we test whether we can predict the behavior of LLM judges using simple features such as part-of-speech tags and find that we can correctly predict 70%-75% of judgments. We conclude this study by analyzing practical use cases, showing that LLM judges consistently detect the on-average better model but largely fail if we use them to improve task performance.

## introduction the automatic evaluation of machine learning models promises to reduce the need for human annotations. specifically, the llm-as-a-judge paradigm @xcite has gained traction, aiming to assess or compare the quality of generated texts automatically. this approach is beneficial for automated data labeling @xcite , selfimprovement of llms @xcite , and ranking the capabilities of llms, potentially concerning specific tasks @xcite . much like judges in the real world, who are expected to be exact, fair, and unbiased (bangalore principles, 2002), 1 code will be made available upon acceptance. provide candidate answers for a math problem, and a judge llm has to decide which one is correct. all three use chain-of-thought (cot) reasoning @xcite . llm judges, should be unbiased and logical. previous works investigate properties and biases of llm judges on generative tasks such as translation or summarization @xcite , typically evaluated using correlation with human annotators, and thus being inherently subjective. in this work, we investigate llm judges on mathematical reasoning datasets. such tasks require complex multi-step reasoning and judgments can be analyzed through the lense of verifiable solutions, allowing us to investigate the relationship between judge and candidate models in a principled manner. in our setup, llm judges are given two answers and they have to classify whether both answers, one of them (which one), or none is correct (see figure 1 . we base our analysis on four large (> 27b parameters) llms and four small (< 10b parameters) llms on three mathematical reasoning datasets. our experiments contain a detailed performance examination. we find that the best tested judge is llama 3.1 70b, reaching 60% to 90% judgment performance, depending on the dataset. our results confirm the intuition that judgment performance is aligned with task difficulty. we perform a statistical analysis of judgment performance and model quality. we find that the individual task performances of judge and candidate models are highly indicative features of judgment performance, as they explain most of the variance in a linear model (as measured by r 2 ). on the subset of questions where the candidate models give one correct and one incorrect answer, we uncover an intriguing correlation between judge performance and candidate models' task performance, indicating that llm judges tend to select incorrect answers from better models. we hypothesize that judges partially base their judgment on linguistic cues rather than solely on the reasoning withinin the answers. we follow literature analysing machine-generated text @xcite and find that 70%-75% of the judgments can be predicted using simple linguistic features, highlighting the systematicity behind the judge decisions. lastly, we analyze practical use cases and discuss usage recommendations. our experiments suggest that llm judges reliably detect the model of higher task performance but can not reliably improve task performance. rather, we find that it is more sensible to use the judge model as an answer generator, and subsequently take the majority vote of all three answers. in summary, our contributions are as follows: ## general setup in the following, we describe the problem setting, including the used notation, and the general experimental setting including used models and datasets. ## performance analysis the experiments have multiple degrees of freedom, such as judges, candidate models, and datasets. to gain a comprehensive understanding of judges' behavior, we consider two perspectives. first, we investigate judge performance for each dataset, aiming to associate judge performance with task difficulty. second, for a fixed dataset, we analyze judge performance across different pairs of candidate models. ## population-level analysis: judgements and model quality in this section, we investigate the relationship between llm judgments and candidate llm quality. first, we provide a statistical analysis where we use llm task performance to explain the variance in llm judgment performance. further, we focus on the subsets where the candidate models make exactly one correct and one incorrect prediction. we observe a strong statistical relationship between the difference in candidate task performances and judgment performances. a good indicator of the competence of a model on a specific dataset is its task performance. clearly, there is a relationship between the quality of the involved models and the made judgments. we investigate the relationship between task performances (of candidate and judge models) and judgment performance. setup. we fit multiple different linear regression models using the judgment performances as the target variables y , including all variations of judges, model pairs (a, b), and datasets d. regarding the covariates x in the model, we solely use the task performances s x , @xmath0, a, b} of judge and candidate models, to predict judgment performance. since we are not specifically interested in the individual features' effects, but rather in their ability to explain the variation of judgment performance, we rely on the coefficient of determination, r 2 , for evaluation @xcite , see appendix d). results. the results are shown in ## sample-level analysis: judgments and stylistic patterns in section 5, we found that the quality of a candidate llm (as indicated by the task performance) correlates with the made judgment. we hypothesize that models of higher quality exhibit a particular style of expressing themselves and judges partially base their judgment on the incorporated textual cues. motivated by recent work in machinegenerated text detection which finds that llms often exhibit certain styles @xcite or patterns @xcite , we aim to gain a deeper understanding of whether shallow or even contentindependent patterns affect the final judgment. setup. we separate all judgments each judge made into training and test splits and train two classifiers. the test accuracy is reported in ## usage recommendations lastly, we aim to give some usage recommendations. we start by analyzing two applied questions, namely, whether llm judges can identify models of higher task performance and whether llms should be used to improve task performance. in the end, we discuss those results, connecting them to the overall insights of this paper. ## conclusion we conduct a thorough analysis of llm judges on mathematical reasoning tasks. we evaluate the judgment performance of eight models of different sizes on three datasets. we find that larger judge models generally outperform smaller judge models and that judges can reliably detect whether both answers are correct. our analysis reveals a strong correlation between judgments and task performance, indicating that judges tend to choose models of higher quality even if their answers are incorrect. we hypothesize that llm judges partially base their decisions on linguistic cues in contrast to the reasoning within the answers. we support this hypothesis with our experiments showing that 70% of the judges' decisions can be predicted using simple linguistic features such as n-grams of partof-speech tags. lastly, our analysis finds that llm judges reliably detect llms of higher task performance but are not reliably useable to improve task performance. our results show that llm judges contain biases and suggest that practitioners should not blindly trust llm judges. we advise practitioners to carefully decide whether llm judges should be used in their particular application. with this work, we set the stage for further research to investigate how to understand, use, and improve llm judges.