U - CREAT : Unsupervised Case Retrieval using Events extr A c T ion (2023)

Abstract: The task of Prior Case Retrieval (PCR) in the legal domain is about automatically citing relevant (based on facts and precedence) prior legal cases in a given query case. To further promote research in PCR, in this paper, we propose a new large benchmark (in English) for the PCR task: IL-PCR (Indian Legal Prior Case Retrieval) corpus. Given the complex nature of case relevance and the long size of legal documents, BM25 remains a strong baseline for ranking the cited prior documents. In this work, we explore the role of events in legal case retrieval and propose an unsupervised retrieval method-based pipeline U-CREAT (Unsupervised Case Retrieval using Events Extraction). We find that the proposed unsupervised retrieval method significantly increases performance compared to BM25 and makes retrieval faster by a considerable margin, making it applicable to real-time case retrieval systems. Our proposed system is generic, we show that it generalizes across two different legal systems (Indian and Canadian), and it shows state-of-the-art performance on the benchmarks for both the legal systems (IL-PCR and COLIEE corpora).

## introduction traditionally, in the legal domain, for a given legal case (query document) at hand, lawyers and judges have relied on their expertise and experience to cite relevant past precedents (cited documents). moreover, even when legal professionals have made limited use of technology, it has been mainly restricted to boolean queries and keywords. however, as cases increase, it becomes difficult for even experienced legal professionals to cite older precedents. nlp-based technologies can aid legal professionals in this regard. the task of prior case retrieval (pcr) has been formulated to address this problem @xcite . more concretely, the task @xmath0, "body send examination", "<@xmath1, "there be injury", "injury find body", "there be evidence" candidate doc events query doc events matching @xmath2. pcr can be particularly important in populous countries like india, where the number of cases has been growing exponentially, for example, there are 41 million pending cases in india @xcite . technologybased solutions such as pcr can make the process streamlined and efficient, expediting case disposal. pcr is different from standard document retrieval tasks. it is primarily due to the nature of legal documents themselves. legal documents, in general, are quite long (tens to hundreds of pages), which makes each document in both the query and candidate pool long. legal documents are unstructured and sometimes noisy (for example, in many common law countries like india, legal documents are manually typed and prone to grammatical and spelling mistakes). moreover, in a common-law system, where the judges can overrule an existing precedence, there is some degree of subjectivity involved, making the task of document processing and retrieval challenging. in this paper, we propose a new large pcr corpus for the indian legal setting referred to as indian legal prior case retrieval (il-pcr) corpus. further, we propose an unsupervised approach for the task of prior case retrieval based on events structure in the document. events are defined in terms of predicate and its corresponding arguments (see figure 1 ) obtained via a syntactic dependency parser. the proposed event-based representation technique performs better than the existing state-of-the-art approaches both in terms of retrieval efficiency as well as inference time. we conjecture that events obtained via a dependency parser play an essential role in providing a short summary of long judgment documents, hence reducing the noise (taskdependent non-relevant information) by a considerable margin (also shown in fig. 2 ). the focus of this paper is an unsupervised and fast approach for retrieving relevant legal documents, in contrast to resource and compute-intensive supervised approaches. in the legal domain, supervised algorithms often require hand-crafted engineering/tuning with considerable experimentation to enable deployment in a real-time scenario, making them harder to adapt to an industrial setting. although not a fair comparison, our proposed method shows an improvement of 5.27 f1 score over a recent state-of-the-art supervised method @xcite for the existing pcr benchmark dataset of coliee'21 ( §5.2). in a nutshell, we make the following contributions: ## related work automating processes and tasks in the legal domain has been an active area of research in the nlp and ir community in the past few years. for example, several tasks/research problems and solutions have been proposed, e.g., catchphrase extraction @xcite , crime classification @xcite , summarization @xcite , rhetorical role prediction @xcite and judgment prediction @xcite @xcite @xcite @xcite @xcite . some earlier works @xcite in prior case retrieval have used feature-based machine learning models such as svm. since the past few years, the competition on legal information extraction and entailment (coliee) has been organized annually @xcite . coliee has spurred research in pcr. researchers participating at coliee have shown that bm-25 based method is a strong baseline. most of the participating systems in col-iee have used models based on bm-25 combined with other techniques like tf-idf, language models, transformers, and xg-boost (e.g., @xcite @xcite @xcite @xcite ). citation network-based approaches @xcite @xcite @xcite are not meaningfully applicable to pcr as the legal citation networks are quite sparse. @xcite proposed bert-based query-by-document retrieval method with multi-task optimization. we also experimented with transformer-based methods for retrieving prior cases as described in §5.1. in the nlp community, researchers have used event-based information for many different natural language understanding (nlu), and commonsense reasoning tasks @xcite @xcite @xcite . for example, @xcite extracted events from a document and used the event-centric graph representation for information retrieval and multi-document summarization tasks, where they define an event as a tuple of predicate (action) and corresponding arguments (participants/actors). in the legal-nlp domain, event-based representations have not been explored much, as also pointed out in the survey by @xcite . in this work, we employ event-based representation for pcr. ## il-pcr corpus and pcr task to spur research in the area of pcr, we propose the creation of a new corpus for the task of pcr: indian legal prior case retrieval (il-pcr) corpus. il-pcr corpus is a corpus of indian legal documents in english containing 7070 legal documents. ## event based representations a story or an incident is best described in terms of a sequence of events @xcite @xcite . if we consider a case judgment document to be a narrative about how things (e.g., situations in the form of facts) developed, then it is best to represent a legal document in terms of events. we define an event as a tuple containing predicate (describing the main action, typically it is verb/verb-compound) and its main arguments (describing main actors/participants, typically these correspond to subject, object, indirect object, and prepositional object) as shown in fig. 1 . ## experiments, results and analysis datasets: we experimented with the coliee-21 and il-pcr corpora. since the two corpora are different, it enables checking the generalization capabilities of models. evaluation metric: we use a micro-averaged f1 score as the evaluation metric (as done in coliee-21 foot_0 ). in practice, models predict a relevance score for each candidate for a given query. top-kranked candidates are considered for prediction (i.e., whether a candidate is cited or not). as done in previous work @xcite , we select k based on the best performance on the validation set and report the f1 on the test set using the same k value (metric definition is provided in app. a). ## conclusion in this paper, we proposed a new large dataset (il-pcr) for prior case retrieval and the u-creat pipeline for performing event-based retrieval for legal documents. we ran a battery of experiments with different types of models to show that event-based methods have better performance and much better inference times (and hence amenable to production settings) compared to existing unsupervised approaches and some of the supervised approaches (e.g., ∼ 5.27 f1 score improvement on coliee) on two completely different datasets. in the future, we plan to combine event-based methods with supervised techniques such as contrastive learning to develop more efficient models.