STARD : A C hinese Statute Retrieval Dataset Derived from Real-life Queries by Non-professionals (2024)

Abstract: Statute retrieval aims to find relevant statutory articles for specific queries. This process is the basis of a wide range of legal applications such as legal advice, automated judicial decisions, legal document drafting, etc. Existing statute retrieval benchmarks emphasize formal and professional queries from sources like bar exams and legal case documents, thereby neglecting non-professional queries from the general public, which often lack precise legal terminology and references. To address this gap, we introduce the STAtute Retrieval Dataset (STARD), a Chinese dataset comprising 1,543 query cases collected from real-world legal consultations and 55,348 candidate statutory articles. Unlike existing statute retrieval datasets, which primarily focus on professional legal queries, STARD captures the complexity and diversity of real queries from the general public. Through a comprehensive evaluation of various retrieval baselines, we reveal that existing retrieval approaches all fall short of these real queries issued by non-professional users. The best method only achieves a Recall@100 of 0.907, suggesting the necessity for further exploration and additional research in this area.

## introduction statutes are written laws formally created and approved by a legislative body, such as a parliament or congress @xcite . they set out specific rules and guidelines within a certain area or jurisdiction. therefore, statutes are the primary source of legal authority in civil law countries and also play a significant role in common law jurisdictions. statute retrieval involves finding relevant statutory articles or sections of laws for a specific query. this process is vital in the legal field and supports a wide range of applications, including legal advice services, automated judicial decisions, and logical legal analysis. this task is challenging for the following reasons: (1) statutes use complex legal terminology and linguistic structures rarely found in open-domain corpus. as a result, traditional retrieval models that lack domain-specific knowledge may struggle to capture the meanings of these specialized terms accurately. (2) the criteria for assessing information relevance in the legal domain differ greatly from those used in open-domain search tasks. general search tasks focus mainly on textual similarity, while legal tasks involve legal reasoning that requires the understanding of different areas of law, the relations between them, as well as the relevance of specific legal principles and their practical applications. due to the challenging nature of statute retrieval and its paramount importance in civil law systems, significant progress has been made in this field. for example, the annual coliee competitions introduce a series of statute retrieval tasks using the questions extracted from the japanese legal bar exams @xcite @xcite . these tasks aim to retrieve relevant statute law from the japanese civil code article according to the question from bar exams. aila @xcite competitions also introduce a series of statute retrieval datasets. the queries from aila are case documents that were judged by the supreme court of india. the candidate statutes are part of the set of statutes from indian law. despite these advancements, a significant gap persists in addressing real queries from nonprofessional people, who represent a large population of legal advice service users. the current statute retrieval benchmarks are primarily based on queries from formal legal documents, such as bar exam questions or supreme court case documents, which differ significantly from the everyday language used by the general public. however, queries from non-professionals often lack precise legal terminology and may include ambiguous references to legal concepts, which significantly complicate the task of statute retrieval. to address the limitations of existing benchmarks, we propose statute retrieval dataset (stard) i.e. stard, a chinese statute retrieval dataset based on real legal consultation questions from the general public. the stard dataset comprises 1,543 query cases collected from real-world legal consultations and 55,348 candidate statutory articles extracted from all official chinese legal regulations and judicial interpretations. we conduct experiments on a wide range of information retrieval (ir) baselines on the stard dataset, including traditional lexical matching models, open-domain neural retrieval models, legal domain neural retrieval models, and a dense retriever trained with data annotated by gpt-4. the experimental results show that all existing baselines fall short of accurately and comprehensively retrieving the relevant statutes, leaving significant room for future work. additionally, our experimental results show that employing stard as an external knowl-edge source for retrieval-augmented generation (rag) significantly enhances the performance of large generative language models (llms) on legal tasks. this indicates that stard is useful for developing more accessible and efficient legal systems. in conclusion, the contributions of this paper are as follows: ## annotation framework this section explains how annotators transform general questions into professional legal questions submitted by non-professionals and then identify the most relevant legal statutes to support these questions. to be specific, annotators use a threestep method: recall, query decomposition, and filtering (illustrated in figure 1 ). this method mirrors the structured approach commonly used in legal reasoning, which involves three logical steps: establishing a broad legal principle (major premise), applying it to the specific facts of a case (minor premise), and then concluding. this section is organized into three subsections, each detailing a part of the annotation process that is designed to mirror these logical steps in legal reasoning. ## dataset statistics and analysis the basic statistics of our proposed dataset are shown in figure 2 presents the distribution of queries across the number of relevant statutory articles, highlighting the varied complexity within the dataset. a substantial majority of the queries, 843 out of 1,543, correspond to just one relevant statutory article, indicating a significant number of queries can be addressed with a single, specific legal reference. this could suggest that many of the non-professional queries are focused and pertain to specific legal issues that require straightforward statute retrieval. however, 45% of queries require multiple statutory articles which indicates some of the questions are more complex, involving multiple references of law. this diversity in query complexity demonstrates that our dataset is capable of accommodating a wide range of legal questions, from straightforward to highly intricate. ## statute retrieval experiment 6.1 selected retrieval baselines we consider four types of baselines for comparison, including traditional ir methods, pre-trained language models on general domain data, plms tailored for ir, and pre-trained language models built with legal documents. the implementation details of these baselines are provided in appendix b ## conclusion we present stard, a new benchmark consisting of 1,543 questions from the general public. to the best of our knowledge, stard is the first chinese statutes retrieval dataset tailored for the general public. moreover, we propose an annotation framework to improve the accuracy and relevance of statute retrieval annotation, which offers valuable guidelines for future legal annotations. our experiments across various retrieval models highlighted the complexities of non-professional statute retrieval, indicating the necessity for further exploration. additionally, we demonstrated that integrating the stard dataset significantly boosts the performance of llms in legal tasks, showcasing its potential to enhance legal ai applications.