A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences (2025)

Abstract: While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose TL agent that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law-reasoning in the “Intelligent Court”.

## introduction in recent times, artificial intelligence (ai) has demonstrated a profound impact on legal applications, including the generation of legal document summarization @xcite , argument mining, @xcite and legal case retrieval @xcite . while recent advances focus on generating impartial and interpretable judicial judgments based on established criminal fact (t.y.s.s. @xcite @xcite . however, the premise for ensuring this process is the accurate determination of the ultimate criminal facts. the fundamental challenge remains: how to construct logically rigorous, evidence-backed ultimate criminal facts from evidentiary materials and inferred interim facts. accurate criminal fact determination forms the cornerstone of judicial fairness @xcite ; @xcite ; chafee @xcite . however, evidence i recovered i saw my husband scuffling with a man. the man who i know as freddie bywaters was running away. case description: "when we got near endsleigh gardens a man rushed out from the gardens and knocked me and pushed me away from my husband. i was dazed for a moment. when i recovered i saw my husband scuffling with a man. the man who i know as freddie bywaters was running away. he was wearing a blue overcoat and a grey hat. edith first fell in love with bywaters in september, 1921, and that she had told her husband that she had given him cause for divorce. ## • experiences. human experience e used during connecting evidence v and fact f, and forming coarse factum probandum. practitioners or intelligent models may need personal experiences for reasoning. the experiences help to explain why the decision maker inference like this, making the process more explicit to understand. ## dataset construction we construct a high-quality dataset of case descriptions with multiple levels of annotated factum probandum, evidence and links with correlative factum probandum, and the involved experiences, which follow our schema and show the explicit path of the law reasoning process. this section case description: it is necessary, as shortly as possible, to review some of the facts of this essentially commonplace and unedifying case. the appellant, edith jessie thompson, is twenty-nine years of age. she is the daughter of a mr. graydon, and seven years ago she married mr. percy thompson, the man now dead, the only person who in this case excites any sympathy… input input output interim factum probandum:edith incited freddie to murder her husband. evidence:i wrote to freddie, and were written to him without my husband's consent. when he was at home in england, we were in the habit of going out occasionally together without my husband's knowledge. can the interim factum probandum be inferred from the evidence? true figure 6: illustration of the evidence reasoning in sub-task2. case description: it is necessary, as shortly as possible, to review some of the facts of this essentially commonplace and unedifying case. the appellant, edith jessie thompson, is twenty-nine years of age. she is the daughter of a mr. graydon, and seven years ago she married mr. percy thompson, the man now dead, the only person who in this case excites any sympathy… input input output interim factum probandum:edith incited freddie to murder her husband. evidence:i wrote to freddie, and were written to him without my husband's consent. when he was at home in england, we were in the habit of going out occasionally together without my husband's knowledge. did experience help in the reasoning? yes what the experience is used in the reasoning? (a) older women tend to dominate, (b) older women are insecure. delves into the details of the crowd-sourcing construction, statistical analysis and quality control(in section b and section c). we utilize publicly available data and implement strict controls over the annotation process. you can get more details on bias and ethical considerations in section d from the appendix. ## approach for the task, we propose our knowledge-enhanced transparent law reasoning agent (tl agent). this approach, which see the whole law reasoning process as a tree structure, adheres to the established analytical approach employed by legal professionals. as illustrated in figure 8 , the left side of the diagram depicts the user's input and the agent's output, the middle section outlines the fundamental workflow of the agent, and the right side present the main toolkit of the agent. the task objective and corresponding case information are input into the agent, which, based on the tool manual's instructions, progresses through the stages of thinking, reasoning, and planning. subsequently, the agent selects the appropriate tools from the toolkit to accomplish the task. to enhance the quality of the outcome, the agent analyzes the returned results to determine if additional tools are necessary to improve accuracy. this process continues until the agent deems the results satisfactory, at which point it invokes the finish tool to conclude the task. in the following sections, we will delve into the details of our agent toolkit in section 4.1 and agent strategy in section 4.2. the prompt design details of each tool can be referred to in section e in the appendix. ## experiments our agent has been rigorously compared against several classic llms available on the market, as detailed in section 5.2. additionally, we conducted a comprehensive comparison with state-of-the-art reasoning models, including o1 and r1, as discussed in section f. to further validate the effectiveness of our tools, we performed an ablation study, the results of which are presented in section g. these comparisons and analyses collectively demonstrate the robustness and superior performance of our agent in various tasks and scenarios. ## conclusion artificial intelligence legal systems currently face challenges in law reasoning. to address this issue, we propose tl agent for law reasoning. by following two key steps: schema design and establishing tree-organized structures process (i.e., evidential reasoning), we can develop an abstract, systematic, and formalized reasoning process for law reasoning tree based on unstructured data. this law reasoning system can serve as a foundation for the advancement of ai legal systems, enabling them to make judgments transparent. to ensure transparency in the judge's decisionmaking process, it is important to visualize the experience relied upon and the intermediate conclusions reached at each step of reasoning and judgment. this serves as a helpful reminder to the judge of which experience was utilized in each step, thereby mitigating the inherent risk of personal bias and enhancing the accuracy of law reasoning along with the final judgment. our contribution in terms of task formulation, dataset and modeling pave the way for transparent and accountable ai-assisted law reasoning.