L e P a RD : A Large-Scale Dataset of Judicial Citations to Precedent (2024)

Abstract: We present the Legal Passage Retrieval Dataset, LePaRD. LePaRD contains millions of examples of U.S. federal judges citing precedent in context. The dataset aims to facilitate work on legal passage retrieval, a challenging practice-oriented legal retrieval and reasoning task. Legal passage retrieval seeks to predict relevant passages from precedential court decisions given the context of a legal argument. We extensively evaluate various approaches on LePaRD, and find that classification-based retrieval appears to work best. Our best models only achieve a recall of 59% when trained on data corresponding to the 10,000 most-cited passages, underscoring the difficulty of legal passage retrieval. By publishing LePaRD, we provide a large-scale and high quality resource to foster further research on legal passage retrieval. We hope that research on this practice-oriented NLP task will help expand access to justice by reducing the burden associated with legal research via computational assistance. Warning: Extracts from judicial opinions may contain offensive language.

## introduction a third of the global population lives in a common law jurisdiction where legal arguments are based on prior decisions, known as precedent @xcite . judges and lawyers use citations to build on precedent and frequently quote passages directly from prior cases. the u.s. legal system is an example of a common law system and u.s. federal courts have produced around 1.7 million published judicial opinions, giving rise to tens of millions of passages containing legal rules, standards, and explanations, which could potentially be cited in new cases. lawyers and judges frequently cite such passages as the basis for their arguments (a prototypical example is shown in figure 1 ). as a re-sult, identifying appropriate precedent relevant to a given argument represents a fundamental component of legal practice. this is a complicated and time consuming endeavour: based on the case law access project, a public repository of u.s. case law, there are almost 2 million published federal judicial opinions with an average length of around 4,500 tokens. the sheer volume of passages which could potentially be cited thus adds to the complexity of legal research, which is exacerbated by subtle rules about the contexts in which a given passage is legally binding. we provide the large-scale dataset, lepard, which can be used to develop computational retrieval methods that facilitate the retrieval of u.s. federal court precedent. lepard was constructed by focusing on how judges actually used precedential passages and as such it builds on millions of expert decisions. in practice, highly paid attorneys spend significant time on legal research to find relevant precedent-and they routinely bill up to @xmath0100 per search (franklin county law library, 2023). legal retrieval has been explored in some prior work. @xcite introduces the legal passage retrieval task, however, no corresponding dataset was released and the paper focused on just 5,000 target passages (in contrast to 1.8 million in lep-ard). this is a general problem in legal nlp where large-scale professionally annotated data sources remain proprietary foot_0 . moreover, creating such resources remains costly due to the intricacies of legal language, which complicate the creation of largescale resources without expert annotators who tend to be very costly. the lack of data has in turn made it challenging for legal nlp research to focus on tasks aligned with the needs of legal practitioners. other related work includes the coliee shared task series related to legal case retrieval (e.g., @xcite . in this setting, a system is given a query and has to retrieve the most related case (or statute) from a pre-defined knowledge base. compared to these information retrieval tasks using synthetic queries, our dataset construction is more closely aligned with actual legal practice. furthermore, the coliee datasets remain limited in size, containing around 4,400 cases which could potentially be retrieved 2 , whereas our dataset allows us to investigate legal passage retrieval methods at scale, containing the universe of all cited legal passages in u.s. federal courts. this setting more closely resembles how a practicing attorney would perform legal research. finally, lexical overlap seems to play a significant role in col-iee datasets @xcite , making bm25 a strong baseline in that setting. in contrast, we find that this is not the case for lepard. a growing body of work investigates legal citation prediction @xcite or the retrieval of relevant cases given a query @xcite . based on the preceding context from a legal document, the goal in legal citation prediction is to identify the citation that supports the context in question. by contrast, in legal passage retrieval, the aim is to identify a specific passage of precedent rather than a citation to a whole case (which is usually tens or even hundreds of pages long). we believe there are several reasons to focus on legal passage retrieval over legal citation prediction. legal citation prediction accuracy numbers seem very strong (see e.g., @xcite . we attribute these results to the long-tailed distribution of citations and believe that models take shortcuts to determine a topic for a snippet and then return the most cited cases for these topics-whereas legal passage retrieval inherently requires more involved legal reasoning. this also connects to relevance in legal search, i.e., finding the appropriate target @xcite . we believe legal relevance is more strongly captured by searching for short passages, rather than predicting citations to entire cases, because a case is likely to deal with multiple independent arguments. some passages may not be semantically linked to the concepts they stand for, making it difficult to identify them using lexical overlap or semantic search. 3 instead, the link is established via frequent citations. by contrast, sometimes there exists an entailment relation (see e.g. @xcite between the context and the cited source passage, where the two passages are connected via legal reasoning. however, we find that this entailment in legal reasoning manifests differently in practical legal settings than in other nlp contexts. thus, models trained on e.g, natural language inference @xcite fail to recognize such relations in lepard. hence, our specially curated dataset may better facilitate the approximation of legal reasoning by nlp models. finally, from the perspective of practitioners, we believe that it is more useful to predict specific passages than citations to cases that may be hundreds of pages long. ## legal passage retrieval dataset (lepard) u.s. federal courts are bound by the doctrine of stare decisis, which means that they must abide by past decisions. as a result, judges and lawyers build their arguments on citations to precedent. often these citations will be accompanied by quotations. when performing legal research, frequently cited passages of precedent are often displayed prominently by research platforms (known as "headnotes" or "key cites") and serve as quasisummaries of judicial opinions. in this work, we leverage the quotations contained in judicial opinions to assemble a large dataset of precedential passages. ## dataset statistics quotations serve several purposes in legal writing: they may be used for emphasis, refer to case documents and exhibits, introduce information from witness or expert testimony, cite supporting materials like treatises or academic publications, or they may reference precedential court opinions. across all federal judicial opinions, we identify 15.8 million quotations over five words long and we successfully match 4.3 million of these to precedential passages. while quotations and citations in judicial opinions offer several interesting avenues for legal nlp and legal passage retrieval, we focus on quotations that can be mapped to single sentences from another opinion. future work could also examine the retrieval of longer passages or move beyond quotations to general citations (many of which are associated with a "pincite" or page number). in this section, we present several summary statistics about lepard (see second, the sentence lengths vary substantially and this results in passages and contexts of varying lengths (the longest passage is over 18,000 characters long). this means that many passages and contexts will be truncated by standard text retrieval approaches. third, most destination opinions contain several passages (around 7 on average, but occasion-ally tens or hundreds). this suggests that there are multiple contexts that occur within a single opinion-something that will be familiar to legal practitioners. in our view, this validates the approach of using local context before a quotation rather than searching for more remote context that may be less relevant (for example, many opinions will discuss factors related to jurisdiction or venue early on but these will not come up anywhere else in the opinion). fourth, the average source opinion is represented 11 times in our data. while we treat passages from the same source as separate, it appears likely that they would be conceptually linked (since the portions of an opinion that are cited tend to be somewhat novel or unique and it is uncommon, though not impossible, for there to be multiple such passages in the same opinion). future work could thus explore whether passage retrieval benefits from grouping passages by their source. finally, we find that there is a tremendous amount of variance in the training data by source court. we include courts to allow future users of lepard to narrow predictions by court in order to consider the role of binding precedent. however, it appears that for most courts, there is insufficient data to train independent models. experimental setup. we release three sets of passages mapped to precedent in lepard. in these different sets, we vary the numbers of potential target passages from 10k to 50k (containing the most cited n passages). we also release a dataset with we split the dataset into training, development and test sets, with 90% of the data being in the training set, 5% in the development set, and 5% in the test set. we show dataset statistics in we compare a variety of well-established retrieval algorithms from the nlp literature on lep-ard. these results are intended to serve as a baseline for follow-up work to build upon. our experiments highlight some of the key challenges related to legal passage retrieval and suggest that there is ample room for future work on this task. we specifically experiment with (1) a sparse lexical retrieval approach via bm25 using the anserini package @xcite , (2) a dense embeddingbased retrieval approach using generic sbert embeddings foot_4 @xcite , followed by maximum dot-product similarity retrieval via the faiss package @xcite , (3) a finetuned sbert variant where we fine-tune sbert foot_5 on our training set using the multiple negatives ranking loss @xcite , and (4) passage retrieval as a text classification task where each target passage is mapped to a unique label which is the prediction target for its preceding context @xcite . we provide results in this setting for a distilbert model @xcite , and legal-bert @xcite , a domain-adapted bert model trained on vast amounts of legal documents. the classification models have been trained using the huggingface transformer library @xcite with the standard hyper-parameters found in the trainer class. the aim of our experiments is to include results for established information retrieval methods. these methods have been used extensively in all nlp domains. our experiments are all implemented using their respective libraries and standard hyperparameters described above. ## expert evaluation a legal expert (licensed attorney) reviewed 100 randomly sampled training examples. for each example, the expert determined whether (1) the example was generally clean and free of errors and (2) the preceding context provided sufficient information to determine that the target passage is relevant to the context. based on this evaluation, all examples were clean and free of errors other than preexisting errors stemming from the ocrwe leave addressing these as an opportunity for future work. in 99% of these examples, the expert determined that there was enough information in the context to determine the relevance of the target passage. in the problematic case, the destination context spans two footnotes, the former a series of citations to unrelated memoranda, and the latter an explanatory footnote containing a quotation. due to the cap processing, these unrelated consecutive footnotes appear as adjacent sentences. further investigation showed that this type of explanatory footnote with a quotation is very uncommon in the data. ## discussion we highlight that the legal passage retrieval task is non-trivial, complicated by the long-tailed distribution of cited precedent and the sheer size of the corpus. by publishing lepard, we aim to encourage nlp work on a set of problems that are closely aligned with the needs of the legal profession. more broadly, our aim is to offer an example of how nlp can be used to broaden access to justice and to catalyze similar work in other legal domains. one of the challenges of legal research is that not all case law content carries the same weight. on the one hand, the structure of court systems means that precedent that is binding in one court may not be binding in another court, even if they are part of the same system (e.g., precedent from the u.s. district court for the district of massachusetts is not binding in the u.s district court for the district of oregon because these district courts are part of different judicial circuits within the u.s. federal judiciary). similarly, old precedent may be overturned and thus lawyers must be careful to cite "good law" (although we find that passages tend to be cited for an average of about ten years, see appendix b). on the other hand, not everything that is said in a judicial opinion has the status of precedent: only the elements of a court's reasoning that are essential to the decision bind future courts while other content contained in a judicial opinion is known as obiter dictum and is not legally binding. as a result, methods that focus on lexical overlap or semantic search create a large risk of retrieving content that is not binding precedent. lepard addresses these issues in two ways. first, we include the court and date associated with each precedent to facilitate the identification of precedent that is binding in a certain court and time. second, only passages that have been previously cited by judges are included in the dataset, which significantly reduces the probability of retrieving non-binding dicta. while we note that requiring a passage to be cited at least once restricts our dataset, we believe this limitation is far outweighed by the value of knowing that the passage has been selected for citation by a federal judge. one particularly promising application of precedent prediction is its potential to serve as the basis for retrieval augmented generation using large language models (rag). rag has been put forward as a method of allowing models to generate text based on information that is not contained in the training data @xcite @xcite . in the context of legal research and writing, rag appears to have several key advantages. first, rag is likely to increase the correctness of citations by allowing practitioners to ensure that only real precedent are cited (i.e., reducing, though not eliminating, the risk of hallucinations), the cited precedent is relevant to the particular court, and the cited precedent remains good law (it has not been overturned). the importance of this capability was highlighted by the recent mata v. avianca airlines case where an attorney relied on chatgpt to write a brief that turned out to rely on non-existent references @xcite second, rag is more easily updatable than finetuned models and thus allows case law to be quickly updated as new cases come out and old cases are overturned @xcite . third, rag is auditable in the sense that practitioners see the basis for generated outputs, allowing them to remove any irrelevant precedent before text is generated. the effective design of such systems to integrate well into lawyers' exist workflow raises interesting questions around human-computer-interaction. while rules of professional responsibility related to lawyers' use of generative ai continue to evolve, some proposals highlight an attorney's "duty to supervise" the technologies they use @xcite and the ability to evaluate what precedent will be used as a basis for a brief appears to be a likely prerequisite for "supervising" brief writing models. ## conclusion we introduce lepard, a large-scale dataset for predicting a target precedential passage given a legal argument context. legal passage retrieval is an important task for legal practitioners, and a challenging nlp retrieval task. from a legal perspective, searching for relevant case law consumes significant resources and contributes to the cost of litigation and the associated access to justice gap. from an nlp perspective, legal passage retrieval is a retrieval task with little lexical overlap between queries and targets, which makes it a particularly interesting retrieval problem. we present various experiments to provide initial benchmarks and to highlight the difficulty of the legal passage retrieval task. there are several approaches toward better legal precedent retrieval, some of which we outline here, and the experiments we present are intended as baselines rather than optimal solutions. one example approach is to combine citation and passage retrieval to first find relevant cases and then identify specific passages within them-which can be thought of as a retrieve and re-rank approach. alternatively, one could also retrieve the top-n passages, and re-rank those with a more powerful re-ranker. we are excited for lepard to serve as a large-scale resource for such experiments and other retrieval research in the legal domain.