Legal Case Retrieval: A Survey of the State of the Art (2024)

Abstract: Recent years have seen increasing attention on Legal Case Retrieval (LCR), a key task in the area of Legal AI that concerns the retrieval of cases from a large legal database of historical cases that are similar to a given query. This paper presents a survey of the major milestones made in LCR research, targeting researchers who are finding their way into the field and seek a brief account of the relevant datasets and the recent neural models and their performances.

## introduction legal case retrieval (lcr) is the task of retrieving cases from a large legal database of historical cases that are similar to a given query case @xcite @xcite @xcite @xcite . while a query case is composed of the factual description of the event underlying the case, each historical case in the legal database is stored as a legal judgment document that describes not only the facts of the case, but also the court's decision and the reasoning involved in the decision process. figure 1 shows two examples of a query case and a historical case. regardless of the legal system adopted by a country, lcr offers significant practice value. in the common law system (such as the u.s.), precedents (prior cases decided in the court) are cited by legal professionals (e.g., lawyers, judges) to support their arguments for their intended outcome in the current case @xcite . even in the civil law system (such as china and germany), where statutes are cited as arguments, lcr still provides crucial reference information (such as the statute(s) applicable to the historical case and the court's decision) for both legal professionals and laymen in need of legal advice. since the first paper on lcr was published in 1994 @xcite , lcr has remained an active area of research in the information retrieval (ir) community. note that lcr can naturally be recast as a standard ir task, with the goal of returning a list of historical cases ranked by their relevance to the query case. in fact, this is how early lcr systems were built: a traditional retrieval model (e.g., tf-idf) is used to measure the relevance of each historical case to a query case and impose a ranking on the historical cases based on the resulting similarity values. not surprisingly, like standard ir models, these lcr systems can efficiently rank a large number of documents @xcite . however, their accuracy is fairly poor. this should not be surprising either, as they did not attempt to address the unique challenges associated with lcr that arise primarily from the fact that the definition of relevance in the legal domain is not identical to the typical definition of text relevance (e.g., topic relevance, semantic relevance) @xcite . to exemplify, consider again the two cases in figure 1 @xcite . they appear to be similar as both are theft events where the item involved is antibiotics and the total values differ only by @xmath0908 vs. @xmath1908 worth of antibiotics. historical case: @xcite respondent rxx entered into a walgreens pharmacy without permission and stole $998 worth of antibiotics. historical case: @xcite respondent jxx had a grudge against rxx. jxx stole rxx's heart medicine, which caused the victim to die of a heart attack. after that, jxx fled abroad. while lcr has primarily been tackled in the ir community, we believe that this task would also be of interest to nlp researchers since lcr involves information extraction from text data. our goal in this paper is to increase the awareness of this task among nlp researchers and stimulate their interest in this task, especially since it is far from being solved even after nearly 30 years of research. foot_0 ## modeling challenges and issues for lcr systems to be successfully deployed in practice, not only should they be accurate and efficient, but they should meet the expectation of their target users (i.e., legal professionals and laymen who seek legal advice). with this goal in mind, we discuss seven modeling challenges for lcr researchers. the first four concern accuracy, the fifth one concern efficiency, and the last two concern improving user confidence and satisfaction. ## corpora in this section, we present six corpora that have been widely used for training and evaluating lcr systems. ## evaluation metrics existing lcr models are either classification models, which determine whether a query case and a candidate case are similar, or ranking models, which rank a set of candidate cases in terms of their relevance to a query case. several metrics have been developed to evaluate lcr models. to evaluate ranking-based models, the metrics used include accuracy@k, precision@k, recall@k, f1@k, ndcg@k (normalized discounted cumulative gain @xcite ), and map (mean average precision (tran et al., 2019)), all of which evaluate performance based on the top k cases retrieved by a model @xcite @xcite . to evaluate classification-based models, the metrics used include accuracy as well as microaveraged precision, recall, and f1 @xcite @xcite @xcite . to facilitate the comparison of a classification model and a ranking model, researchers have (1) computed accuracy@k, precision@k, recall@k, and f1@k values of the ranking model by setting k to the number of similar historical cases a query case has on average in the evaluation corpus, and then (2) compared the resulting scores directly with the accuracy, precision, recall, and f1 scores achieved by the classification model @xcite . while accuracy@k, precision@k, recall@k and f1@k can be computed efficiently, they do not consider the order of results and therefore are unable to differentiate models with poor ranking ability from those with better ranking ability. in contrast, ndcg@k and map consider the order of results. compared to map, ndcg@k is more sensitive to rank order because it takes into account the position of the relevant items in the ranked list. ## approaches to lcr in this section, we present an overview of existing approaches to lcr. ## the state of the art in ## ethical considerations several ethical considerations should be taken into account when deploying lcr systems. using lcr systems. lcr systems should be designed with the goal of assisting rather than replacing legal professionals and offering consulting suggestions to people who possess little legal knowledge. like other ai systems, lcr systems are only as good as the data on which they are trained. as mentioned before, bias may exist in the annotations, especially those that are automatically derived, potentially leading to bias in model pre6 compared to other legal tasks such as legal judgment prediction (ljp) @xcite , lcr is a task that is far from being solved. lcr only achieves a maximum of 73.9% in f1 score, while ljp obtains over 90% in f1 score. diction. hence, legal decisions should always be made by professionals rather than lcr systems. debiasing data. the output of lcr systems may affect the court's decisions. as a result, fairness and justice are important principles underlying the development of lcr systems. to avoid having lcr systems make biased predictions, it is important that these systems are trained on data instances that are not biased. for lcr, we believe that there are at least two sources of data biases. first, the data may contain information that should probably not be used when determining similarity (e.g., the name, gender, age, and/or race of the person involved in a case). to ensure that such information is not exploited by lcr systems, the data should be properly anonymized. second, the data may not have been properly annotated. for manually annotated datasets, some human annotators may be racially, sexually, or even politically biased, and hence the annotations they produce could be biased. standard annotation procedures could be used to address this problem. for example, biased annotators could be identified using qualification tests, and biased annotations could be identified if inter-annotator agreement was low. for datasets automatically annotated via citations, citation bias exist. to improve annotation quality, we recommend that a legal expert provide an independent relevance judgment for each data instance. ## concluding remarks in this section, we conclude with several promising directions for lcr research.