CL aw: Benchmarking C hinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis (2025)

Abstract: Large Language Models (LLMs) are increasingly tasked with analyzing legal texts and citing relevant statutes, yet their reliability is often compromised by general pre-training that ingests legal texts without specialized focus, obscuring the true depth of their legal knowledge. This paper introduces CLaw, a novel benchmark specifically engineered to meticulously evaluate LLMs on Chinese legal knowledge and its application in reasoning. CLaw comprises two key components: (1) a comprehensive, fine-grained corpus of all 306 Chinese national statutes, segmented to the subparagraph level and incorporating precise historical revision timesteps for rigorous recall evaluation (64,849 entries), and (2) a challenging set of 254 case-based reasoning instances derived from China Supreme Court curated materials to assess the practical application of legal knowledge. Our empirical evaluation reveals that most contemporary LLMs significantly struggle to faithfully reproduce legal provisions. As accurate retrieval and citation of legal provisions form the basis of legal reasoning, this deficiency critically undermines the reliability of their responses. We contend that achieving trustworthy legal reasoning in LLMs requires a robust synergy of accurate knowledge retrieval—potentially enhanced through supervised fine-tuning (SFT) or retrieval-augmented generation (RAG)—and strong general reasoning capabilities. This work provides an essential benchmark and critical insights for advancing domain-specific LLM reasoning, particularly within the complex legal sphere.

## introduction large language models (llms) have demonstrated remarkable capabilities, approaching expertlevel performance in intricate domains such as coding @xcite and mathematics (glazer @xcite and lexeval @xcite , which often cite statutes imprecisely and omits temporal information, whereas our queries follow the subparagraph-level format used in judicial opinions. (bottom) bertscore of ten llms when asked to recite civil code of the people's republic of china, one of the most frequently used laws. @xcite . this success has catalyzed significant interest in deploying llms in high-stakes fields like medicine @xcite , law @xcite , and economics (axtell and farmer, 2025). however, effective deployment in such specialized areas demands more than superficial understanding; it necessitates genuine mastery of domain-specific knowledge and the capacity for sound reasoning based on that knowledge. a core assertion of this paper is that effective domain-specific reasoning is not merely an extension of general reasoning capabilities. instead, it emerges from a crucial synergy: comprehensive, accurate domain knowledge coupled with robust general thinking abilities. this distinction is pivotal because current llms, typically pre-trained on broad, general corpora, may en-counter and process domain-specific texts (like legal statutes) without achieving the necessary depth of knowledge mastery. consequently, the reliability of their subsequent reasoning remains uncertain, irrespective of their inherent general cognitive skills. this paper investigates two fundamental questions through this lens: (1) do state-of-the-art llms possess reliable, fine-grained knowledge of law, specifically for the precise statute recall essential in judicial practice (the knowledge mastery component)? (2) how effectively can these llms apply this legal knowledge by leveraging their general thinking abilities in practical case analysis (the reasoning application component)? we focus our investigation on chinese law, a domain that presents both a critical societal need for ai assistance and an ideal testbed for evaluating these two distinct yet intertwined components. the sheer volume of litigation in china, with millions of judgments published annually @xcite , highlights the urgent demand for reliable aidriven legal tools. its civil-law system, where judicial decisions primarily hinge on written statutes rather than case precedents, offers a well-defined and structured body of knowledge-a prerequisite for rigorously testing knowledge mastery. furthermore, the moderate pace of amendments to chinese statutes necessitates that llms accurately distinguish between multiple historical versions of the same legal article, providing a stringent test of knowledge precision and currency. such precision is foundational; without robust knowledge mastery, any downstream reasoning, however sophisticated its general approach, will inherently be flawed. despite the critical importance of accurate knowledge recall and its subsequent application, existing benchmarks often categorize recall as a "basic" task @xcite and utilize queries with vague or temporally ambiguous references @xcite ). this overlooks the role of precise knowledge mastery as an independent pillar of domain expertise. moreover, evaluating the practical application of this knowledge-the efficacy of general reasoning operating upon this specialized knowledge base-remains a significant challenge. to address these limitations, we introduce claw, a dual-faceted benchmark meticulously designed to dissect and evaluate these two components. firstly, to assess knowledge mastery, claw features a comprehensive corpus encompassing all 306 chinese national statutes, including every available historical version (424 distinct statute ver-sions in total). this corpus is segmented to the subparagraph level and incorporates precise revision timesteps, enabling fine-grained queries that mirror the citation format used in actual judicial opinions. this results in 64,849 distinct entries for statutory knowledge evaluation. secondly, to assess the application of general thinking abilities to this domain knowledge, claw introduces a casebased reasoning task. this component, built upon 254 representative and challenging cases curated by the china supreme court (complete with judicial reasoning for reference), evaluates how well llms can analyze case descriptions, identify relevant legal issues, and apply statutory knowledge through logical reasoning to derive conclusions. we also propose an llm-as-a-judge pipeline for the faithful and automated evaluation of this reasoning task. our empirical results from the statutory recall benchmark, targeting the knowledge mastery aspect, are concerning. even leading models like the gpt series (o1, gpt-4o, gpt-4.1) frequently cite repealed or hallucinated provisions when queried about chinese law (figure 1 (bottom)). this fundamental deficiency in knowledge inevitably compromises any subsequent attempts at higher-level legal reasoning, regardless of an llm's general cognitive prowess. preliminary analysis of our casebased reasoning task further suggests that both deep domain knowledge and strong general thinking abilities are indispensable for tackling real-world legal tasks. this study underscores that before llms can be reliably deployed in high-stakes fields like law, their domain-specific knowledge mastery must be rigorously assessed and substantially augmented. only then can their general reasoning abilities be effectively leveraged to produce dependable domainspecific applications. our contributions are: legal benchmarks for llms the legal domain's specialized language and complex reasoning necessitate domain-specific benchmarks. key benchmarks include legalbench (english @xcite ), the multilingual lex-treme @xcite , and jurisdictionspecific ones like oab-bench (brazilian portuguese @xcite ). these often cover tasks like legal text classification, summarization, question answering, and to some extent, reasoning. chinese legal llms china's distinct legal system has spurred development of specialized llms and benchmarks. notable benchmarks are law-bench @xcite and lexeval @xcite , which uniquely includes ethical evaluations. leading models like lawyer llama @xcite , chatlaw @xcite , and internlm-law @xcite are trained on diverse chinese legal texts and filtered consultation data, aiming to effectively navigate the local legal landscape. claw builds upon this by providing an even more granular dataset for statutory knowledge and a dedicated component for case-based reasoning. ## conclusion we introduced claw, a novel benchmark meticulously designed to evaluate the depth of chinese legal knowledge and the efficacy of case-based legal reasoning in general-purpose llms. claw's contributions are twofold: a comprehensive, finegrained, and historically versioned corpus of all chinese national laws for precise recall assessment, and a challenging set of reasoning tasks derived from authoritative china supreme court guiding case. our extensive empirical evaluation of ten leading llms reveals a critical deficiency: most models struggle significantly with the accurate recall of specific legal provisions, especially at the granular subparagraph level and under the effects of temporal versioning. this fundamental lack of reliable knowledge mastery severely undermines their potential for trustworthy legal reasoning. our findings from the case-based reasoning task further reinforce the position that effective domainspecific reasoning is not an isolated capability but an emergent property arising from the synergy of deep, accurate domain knowledge and robust general reasoning abilities. while llms may exhibit strong presentational skills, their capacity for rigorous legal analysis and the accurate application of legal principles is often compromised by an insecure knowledge foundation. this resonates with broader challenges in domain-specific ai, where superficial understanding can mask critical gaps in knowledge and true reasoning capacity. we assert that current llms, in their standard form, are not yet equipped for reliable autonomous deployment in many practical legal applications, particularly those requiring high fidelity to specific statutes and nuanced reasoning. the path forward necessitates a dedicated focus on enhancing both knowledge precision and reasoning acuity. this includes strategies such as sft using meticulously curated and structured corpora like claw, the development of more sophisticated and legally-aware rag systems, and targeted research into methods that improve logical inference, the structured application of legal rules, and the verifiability of llmgenerated legal arguments. claw provides an essential resource for the research community to benchmark progress, diagnose weaknesses, and ultimately foster the development of more reliable and trustworthy llms for the legal domain and other specialized fields demanding precision and sound judgment.