C ita L aw: Enhancing LLM with Citations in Legal Domain (2025)

Abstract: In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs’ ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.

## introduction generating responses supported by citations, such as relevant law articles and precedent cases, is essential for ensuring the trustworthiness of large language models (llms) in legal tasks. for laypersons seeking legal advice @xcite , llmgenerated responses grounded in citations provide verifiable information, fostering trust in the system. conversely, for legal practitioners such as lawyers and judges, citations serve as supportive evidence that aids in analyzing complex cases, validating legal arguments, and ensuring decisions align with established legal principles @xcite @xcite . recently, a growing body of benchmark research @xcite has focused on enabling llms to provide citations for the * corresponding author statements they generate. for instance, alce @xcite introduces a benchmark designed to evaluate the ability of llms to generate citationsupported outputs, aiming to improve factual accuracy. webcites @xcite provides a curated database of manually annotated summaries and citations to enhance performance in text summarization and citation generation. while these studies have made notable progress in general domains, they face significant challenges when applied to the legal domain. first, laypersons and legal practitioners interact with llms differently and have distinct expectations for citations. laypersons typically seek legal advice and rely on citations to verify the accuracy of llm responses, whereas legal practitioners pose more complex queries, using llms for legal reasoning, with citations serving as supportive evidence. existing studies fail to address these differences, leading to unsatisfactory performance in real-world applications. second, existing methods often fall short in providing the diverse references required in legal contexts, such as law articles and precedent cases. law articles establish the foundational legal framework, while precedent cases offer concrete examples and interpretive guidance. these two types of references inherently align with the distinct characteristics of civil and common law systems. third, traditional citation evaluation measures, such as rouge @xcite , rely on surface-level similar-case: ...(fact 2)...x1 suggested that we tie him up to force x2 to repay the money...(x2 serious injury)... question: for fact 2, what are the reasons for the view that x1 does not constitute transformative robbery and constitutes intentional injury crime? question: … the school mandatory internship, is it legal to refuse to let the student do the internship on their own? 情景(深蓝)，⾏为(浅蓝)，结果(⻩)，⼤前提(红) [a1]: ''regulations on the…of higher learning article 6" students enjoy the following rights…. ## evaluation citalaw provides a comprehensive evaluation framework incorporating metrics for fluency, correctness, and citation quality. this framework is divided into two levels of analysis: global level and the proposed syllogism level. syllogism, a foundational framework in legal reasoning, comprises three key components: the major premise, the minor premise, and the conclusion. in our legal context, these correspond to the relevant law article or precedent case (major premise), the factual circumstances and actions of a specific case (minor premise), and the resulting legal decision (conclusion). by integrating this syllogistic framework, citalaw goes beyond surfacelevel correctness to evaluate the logical coherence and alignment of llm-generated responses with established legal principles. ## experiments we conduct extensive experiments on our citalaw using the proposed two-level evaluation methods. ## conclution we introduce citalaw, a benchmark designed to explore llms to generate responses with citations in legal scenarios, thus improving the trustworthiness of llms. citalaw includes two categories of questions: laypersons and practitioners. for laypersons, citalaw provides law articles as citations to help them understand the llm's response clearly. for practitioners, both law articles and precedent cases are provided as citations, better supporting their needs for complex reasoning. citalaw offers global-level and syllogism-level metrics and supports the integration of citations into llm inputs to guide generation or using citations to refine llm's response. we conducted extensive experiments on 7 legal-domain llms and 2 popular open-domain llms, providing valuable insights for the deployment of llms in legal scenarios. ## ethical considerations data privacy and confidentiality. the legal datasets used in citalaw include law articles, precedent cases, user questions, and golden responses. these documents were sourced from publicly available databases, ensuring compliance with data privacy and confidentiality standards. we carefully reviewed the datasets to ensure that no personally identifiable information (pii) or sensitive details about individuals were inadvertently included. alignment with legal standards. legal ai systems must align with the ethical and professional standards of the legal domain. our work emphasizes the need for syllogism-based reasoning to ensure logical consistency and adherence to legal principles.