Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation (2025)

Abstract: LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models’ generation and judgment abilities remain inconsistent. We investigate this relationship through systematic dataset- and instance-level analyses across 11 models and 21 diverse tasks. Despite both capabilities relying on the same underlying knowledge, our analyses reveal they are only weakly correlated, primarily due to LLMs’ sensitivity to the responses being judged. To address this, we propose a self-reference-guided evaluation strategy that leverages a model’s own answers as references. This approach significantly strengthens the correlation between generation and judgment abilities, offering a practical path to align these skills and providing a reliable proxy for model selection in evaluation tasks.

## introduction model-based evaluation, which uses large language models (llms) as judges, has gained increasing prominence in natural language processing. this approach, commonly known as llm-as-judge @xcite , has been widely adopted across a range of applications @xcite @xcite . however, a critical question remains: how closely does a model's ability to generate answers correlate with its capacity to evaluate them? prior work has offered divergent perspectives on this issue, with @xcite reporting strong correlation between these abilities and @xcite arguing they are not necessarily aligned. these conflicting findings, based only on dataset-level analyses, highlight the need for more comprehensive investigation. to address this gap, we systematically investigate the relationship between llms' answer generation and evaluation capabilities within the llm-as-judge framework. we focus on answer judgment as the representative evaluation task, as it directly depends on the same knowledge used in answer generation and thus provides a clearer lens for analyzing their correlation. given the widespread use of chain-of-thought (cot) reasoning @xcite , we adopt the cot paradigm for both generation and judgment tasks. furthermore, unlike prior studies that primarily examine datasetlevel correlations, our analysis spans both datasetlevel and instance-level perspectives, enabling a more fine-grained understanding of how these capabilities interact. further analysis in section 5 introduces a self-reference-guided evaluation strategy, which builds upon the reference-guided judging framework @xcite . this approach significantly improves the correlation between a model's answer generation and judgment capabilities, with an average increase of 0.35 across all evaluated cases (as shown in our findings reveal that without special techniques, strong answer generation ability does not necessarily translate to strong judgment ability. the correlation between these two capabilities is generally weak when using standard cot approaches. however, incorporating self-referenceguided judging significantly strengthens this correlation, making generation performance a reliable predictor of judgment capability. these insights offer practical strategies for selecting and utilizing judge models to enhance evaluation performance, effectively aligning generation and judgment capabilities. our contributions are fourfold: capabilities of llm-as-judge. prior work on llm-as-judge has explored their capabilities through benchmarks @xcite @xcite , tuning methods @xcite @xcite , prompting strategies @xcite @xcite , and model interaction architectures @xcite @xcite . several survey papers @xcite have summarized these developments, evaluating various prompt designs @xcite and implementation approaches. ## conclusion in this paper, we conducted a systematic examination of the correlation between performance in answer generation and answer judgment tasks using llms with standard cot prompting. we evaluated 11 widely used llms on 21 benchmark subtasks. our results show that, for most models, performance on answer generation is only weakly correlated with the ability to judge answers, indicating that strong generators are not necessarily reliable judges. in addition, we found that self-referenceguided evaluation methods can strengthen the correlation between generation and judgment capabilities. based on these insights, we offer practical recommendations for selecting models to serve as judges, especially when external references like golden answers or outputs from stronger models are unavailable. in these situations, our work provides an accessible approach to judge model selection by using generation performance as a reliable proxy for judgment capability. this also mitigates the risk of generation and evaluation capabilities diverging as llms continue to develop.