R easoner R ank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking Frameworks (2025)

Abstract: Large Language Models (LLMs) are increasingly adopted across real-world applications, yet traditional evaluations rely on expensive, domain-specific ground-truth labels that are often unavailable or infeasible. We introduce a ground-truth-free evaluation framework focused on reasoning consistency and instruction following, shifting the emphasis from correctness—which is elusive without labels—to transparent, coherent, evidence-based reasoning. Each model response must include a direct answer, a structured multi-step explanation, and supporting evidence, all assessed via semantic similarity and output adherence checks. We further propose TopK-ReRank, which refines rankings by constructing a consensus answer from the most reliable models, reducing ambiguity across diverse reasoning styles. Experiments show that our framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches, providing a more interpretable and efficient alternative for evaluating LLMs in the absence of ground-truth labels.

## introduction large language models (llms) have demonstrated exceptional proficiency in a variety of language-related tasks across multiple domains, and their success has led to rapid adoption in realworld applications @xcite @xcite . this widespread use has spurred a growing interest in understanding and comparing their capabilities @xcite @xcite . traditionally, such evaluations rely on meticulously curated datasets with ground-truth labels to assess the correctness of model outputs. however, obtaining labeled data is expensive, time-consuming, and often requires domain expertise-particularly challeng-ing in specialized areas like healthcare, law, and scientific research. this labeling bottleneck impedes efficient model evaluation and hence slows the integration of new llms into evolving application domains. in many scenarios, labeling is simply infeasible. emerging ("fresh") or zero-shot data can require subjective or multi-faceted responses, making it difficult to define a single "correct" answer. dynamic environments-such as finance, cybersecurity, or breaking news-can also evolve too rapidly for human annotators to keep pace. consequently, researchers urgently need methods to evaluate llms without relying on ground-truth data, a setting that raises fundamental questions about how to gauge "better" or "worse" performance when correctness itself remains elusive. existing labelfree approaches, including llm-as-a-judge @xcite , multi-agent debate @xcite , and majority voting (a.k.a. the most common answer (mienye and sun, 2022)) among peer models, can reduce human labor but introduce new pitfalls. voting schemes may embed biases, and llm-as-a-judge can hallucinate or impose arbitrary standards, thus lacking transparent or reliable criteria for discerning which outputs are truly more coherent, logical, or helpful. in this paper, we propose a new ground-truthfree evaluation framework for comparing llms, centered on reasoning consistency and instructionfollowing ability rather than on correctness-an inherently inaccessible notion without labeled data. our approach is motivated by how humans intuitively evaluate responses in real-world contexts: rather than focusing exclusively on the final answer, we also consider whether the reasoning process is transparent, coherent, and grounded in evidence. building on insights from cognitive science and argumentation theory, we show that logical consistency and clarity of thought often serve as more reliable indicators of model quality compared to merely using a single "correct" label. concretely, we require each model to produce: (i) a direct answer, (ii) a structured, multi-step reasoning explanation, and (iii) supporting evidence. we utilize our proposed semantic similarity-based metric to assess and rank models' reasoning ability, then weight it with instruction-following ability to get the raw "reasoner" scores. finally, we introduce topk-rerank, an adaptive method that generates a refined "consensus answer" through majority agreement among the most reliable models. this step mitigates the ambiguity arising from different reasoning styles, making it easier to identify and reward genuinely coherent and well-supported arguments. thorough experiments and ablation studies demonstrate the effectiveness and robustness of our framework in comparing llms without relying on ground-truth labels. ## conclusions our paper highlights the challenges of ranking large language models in increasingly complex tasks. as dataset difficulty increases, traditional triplet-based ranking methods like selfrank become less reliable, particularly when the performance gap between models widens. in contrast, our proposed reasonerrank method remains stable under these conditions. beyond ranking accuracy, our findings emphasize the need for alternative evaluation criteria when assessing foundational language models, especially in scenarios where ground truth is unavailable or incomplete. even in cases where ground truth exists, these aspects may still be extremely critical in evaluating a model's ability to generalize and maintain consistency across diverse tasks. future work can explore integrating such qualitative evaluation factors into ranking frameworks, moving toward a more holistic assessment of language models. by expanding the scope of evaluation beyond purely performance-driven measures, we can develop more comprehensive frameworks for assessing the effectiveness of large language models in real-world applications.