Are Your LLM s Capable of Stable Reasoning? (2025)

Abstract: The rapid advancement of large language models (LLMs) has shown remarkable progress in complex reasoning tasks. However, a significant disparity exists between benchmark performances and real-world applications. We attribute this gap primarily to current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, especially in complex reasoning tasks where both accuracy and consistency are essential. In this paper, we introduce **G-Pass@** k , a novel evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s performance potential and its stability. Through extensive experiments on various public and newly constructed benchmarks, we employ G-Pass@ k in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency. Our findings reveal a significant opportunity to enhance the realistic reasoning abilities of LLMs, underscoring the necessity for more robust evaluation metrics.

## introduction since the emergence of large language models (llms), complex reasoning, particularly in mathematical problem-solving, has been regarded as the crown jewel of llm capabilities @xcite . numerous models have demonstrated remarkable performance on mathematical tasks, from general-purpose models like @xcite , and the llama series @xcite to specialized models such as @xcite and @xcite , which excel in complex and high-difficulty mathematical reasoning. more recently, long-chain-of-thought (long-cot) models like openai-o1 (openai, 2024b), qwq (team, † corresponding authors. 2024a), and deepseek-r1 @xcite have further advanced the state-of-the-art in mathematical problem-solving. in real-world applications, llms typically employ sampling with predefined decoding parameters (e.g., temperature, top-k, top-p, and repetition penalty) to maintain response diversity. users often regenerate responses or initiate new sessions until obtaining satisfactory answers to specific questions. however, conventional evaluation metrics for llms, such as greedy accuracy, pass@k @xcite , best-of-n (bon), and majority voting, demonstrate significant limitations in measuring real-world performance, particularly regarding long-term consistency. while these metrics effectively capture either instantaneous accuracy or peak performance across multiple samples, they inadequately reflect output stability. such instability poses significant challenges for applications requiring reliable and predictable outcomes, highlighting the need for evaluation metrics that effectively balance response diversity with consistent performance and comprehensive analysis of the reasoning stability of current llms. to address these challenges, we introduce g-pass@k, a novel evaluation metric that simultaneously assesses both reasoning capability and performance consistency. the core principle of g-pass@k lies in evaluating model performance under varying thresholds of correctness, providing a nuanced understanding of model behavior across different stringency levels. by integrating measures of both stability and potential, g-pass@k delivers a comprehensive assessment of an llm's capabilities, particularly valuable for complex reasoning tasks. to thoroughly study the reasoning stability of current llms using g-pass@k, we conduct comprehensive experiments to evaluate various models on multiple mathematical reasoning benchmarks, encompassing different categories of large language models. these include general llms, mathematics-specialized models, and chain-of-thought (cot) enhanced reasoning models, which are known for their step-by-step reasoning capabilities in complex mathematical problem-solving. we also introduce livemathbench, a challenging bilingual mathematics benchmark assembled from multiple competitions and examinations to mitigate the impact of data leakage. through systematic comparison between traditional metrics (greedy accuracy, pass@k) and g-pass@k, our findings reveal distinctive insights into model performance that conventional evaluation approaches may overlook and demonstrate significant limitations in traditional evaluation metrics regarding output stability assessment. we observe a substantial performance degradation across all models as the g-pass@k threshold becomes more stringent, a pattern consistent across both established open-source benchmarks (math @xcite and aime (aime2024, 2024; aime2025, 2025)) and our proposed livemath-bench. specifically, the key observations include: we also investigate the robustness of the newly proposed metric g-pass@k, demonstrating that g-pass@k consistently evaluates model performance across different hyperparameters, guaranteeing its reliability and applicability (section 3.3, section 3.4). furthermore, we conduct a preliminary investigation into enhancing the reasoning stability of the model and identifying the reasons behind strong stability. our investigation reveals two key findings: 1) merely relying on sft does not achieve significant improvement in reasoning stability (section 4.1), and 2) the incorporation of backtracking and self-reflection mechanisms in o1-like models improves the stability of their reasoning capabilities (section 4.2). ## stability of llm reasoning in this section, we perform extensive experiments on the mathematical reasoning task to evaluate and analyze the reasoning stability of current llms, given the prevalence and ease of verification of the mathematical reasoning task. ## enhancing the stability of reasoning in this section, we aim to explore factors that can improve the reasoning stability. initially, we assess whether straightforward supervised fine-tuning on specific corpora enhances reasoning stability. subsequently, we explore the reasons for the pronounced stability of o1-like models, positing that the characteristics of their reasoning paths contribute to this improved stability. ## conclusion in this work, we propose g-pass@k τ and mg-pass@k, novel evaluation metrics that assess both the reasoning capability and performance consistency of llms across varying correctness thresholds. through detailed evaluations conducted on mathematical reasoning benchmarks, we find that current llms struggle with consistent reasoning. additionally, we demonstrate g-pass@k's robustness and preliminarily explore strategies for enhancing the models' reasoning stability.