Evaluating Test-Time Scaling LLM s for Legal Reasoning: O pen AI o1, D eep S eek-R1, and Beyond (2025)

Abstract: Recent advances in test-time scaling of large language models (LLMs), exemplified by DeepSeek-R1 and OpenAI’s o1, show that extending the chain of thought during inference can significantly improve general reasoning performance. However, the impact of this paradigm on legal reasoning remains insufficiently explored. To address this gap, we present the first systematic evaluation of 12 LLMs, including both reasoning-focused and general-purpose models, across 17 Chinese and English legal tasks spanning statutory and case-law traditions. In addition, we curate a bilingual chain-of-thought dataset for legal reasoning through distillation from DeepSeek-R1 and develop Legal-R1, an open-source model specialized for the legal domain. Experimental results show that Legal-R1 delivers competitive performance across diverse tasks. DeepSeek-R1 exhibits clear advantages in Chinese legal reasoning, while OpenAI’s o1 achieves comparable results on English tasks. We further conduct a detailed error analysis, which reveals recurring issues such as outdated legal knowledge, limited capacity for legal interpretation, and susceptibility to factual hallucinations. These findings delineate the main obstacles confronting legal-domain LLMs and suggest promising directions for future research. We release the dataset and model at https://github.com/YinghaoHu/Legal-R1-14B.

## introduction large language models (llms) have recently achieved near-human performance on an increasingly diverse set of benchmarks and application domains @xcite @xcite @xcite . across several flagship llm model families, dedicated reasoning variants, such as openai's foot_0 equal contribution. 2 corresponding authors. o1 (openai, 2024b) and deepseek-r1 (deepseek-ai, 2025) incorporate an explicit internal deliberation phase before producing a final answer. fundamentally, these models extend the chain-of-thought (cot) generated at inference time, thereby allocating increased computational resources per query. the recent open-sourcing of deepseek-r1 further establishes an end-to-end paradigm for training reasoning-centric llms. specifically, deepseek-ai (2025) proposes a four-stage pipeline: (i) coldstart pretraining, (ii) reasoning-oriented reinforcement learning (rl), (iii) rejection sampling-based supervised fine-tuning, and (iv) scenario-wide rl. ## legal-r1 to transfer the reasoning capabilities of deepseek-r1 to the legal domain, we construct a high-quality legal reasoning dataset via rejection sampling guided by deepseek-r1. based on this dataset, we fine-tune the deepseek-r1-distill-qwen-14b, yielding a domain-specific legal reasoning model, legal-r1. ## conclusion this study presents a comprehensive evaluation of 12 llms across 17 chinese and english legal reasoning tasks and introduces legal-r1, an opensource model tailored for legal reasoning. our experiments confirm that test-time scaling improves overall reasoning performance. deepseek-r1 remains among the strongest on both chinese and english tasks, while legal-r1, trained on a curated legal-reasoning dataset, matches or surpasses test-time scaling models on several key tasks and establishes a competitive open-source baseline. error analysis reveals persistent challenges shared by general-purpose and domain-specific models, including outdated or incomplete legal knowledge, misinterpretation of citations, and factual hallucinations. expanding high-quality, up-to-date, multilingual chain-of-thought legal datasets, integrating retrieval or external knowledge bases for fact verification, and developing more robust reasoning architectures will be essential for improving the reliability and practicality of llms in legal reasoning.