C ase E ncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding (2023)

Abstract: Legal case retrieval is a critical process for modern legal information systems. While recent studies have utilized pre-trained language models (PLMs) based on the general domain self-supervised pre-training paradigm to build models for legal case retrieval, there are limitations in using general domain PLMs as backbones. Specifically, these models may not fully capture the underlying legal features in legal case documents. To address this issue, we propose CaseEncoder, a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases. In the data sampling phase, we enhance the quality of the training data by utilizing fine-grained law article information to guide the selection of positive and negative examples. In the pre-training phase, we design legal-specific pre-training tasks that align with the judging criteria of relevant legal cases. Based on these tasks, we introduce an innovative loss function called Biased Circle Loss to enhance the model’s ability to recognize case relevance in fine grains. Experimental results on multiple benchmarks demonstrate that CaseEncoder significantly outperforms both existing general pre-training models and legal-specific pre-training models in zero-shot legal case retrieval. The source code of CaseEncoder can be found at https://github.com/Anonymous-EMNLP2023/CaseEncoder.

## introduction legal case retrieval is a critical process for modern legal information systems, as it aims to find relevant prior cases (i.e., precedents) that serve as important references to the case to be judged. in recent years, pre-trained language modeling (plm) techniques have achieved great success in general-domain retrieval tasks, which has also led to attempts to apply plms in the legal domain. for example, @xcite and @xcite propose a legaloriented plm based on bert @xcite and longformer @xcite , respectively. however, existing legal-oriented plms have limitations in their adaptation to the legal domain, because they mainly rely on replacing generaldomain training data with legal data or extending the input length to fit the long-length characteristic of legal documents. @xcite parse law articles in the form of premise-conclusion pairs to train a multi-level matching network for legal case matching. @xcite propose hier-spcnet which substantially improves the networkbased similarity by introducing legal textual information. however, they still adopt general-domain plms as the foundation of their approach. while interpretable, such models do not fully enable the plm to comprehend legal concepts in case documents, which limits their generalizability and applicability. to address this issue, this paper proposes caseencoder, a plm that leverages fine-grained legal knowledge to improve both the data sampling phase and the pre-training phase. in the data sampling phase, we split law articles into unambiguous branches and construct inner logical relations in each article, which are used to estimate similarity weights between cases. these similarity weights serve as pseudo-labels to guide the selection of positive and negative cases. in this way, our proposed data sampling method improves the quality of sampled cases for further pre-training. in the pre-training phase, caseencoder adopts masked language modeling (mlm) and fine-grained contrastive learning tasks. these tasks aim to match two main concepts in the judging criteria of legal relevance: key circumstances and key elements @xcite , respectively. key circumstances refer to significant case descriptions in the document, while key elements are the legal-level abstraction of key circumstances, which focus more on consistency with law articles. in practice, the mlm task captures the semantic-level case descriptions, while the fine-grained contrastive learning task adopts sampled cases together with their relevance weights to enhance the model's ability to identify key elements. based on the design of the fine-grained contrastive learning task, we propose an innovative loss function, biased circle loss, which leverages the obtained fine-grained relevance score to optimize the recognition of key elements. there are two advantages of annotating law articles as fine-grained legal knowledge. 1) cost efficiency: law articles are usually close sets with a limited number of documents regardless of the legal system, and the number of law articles is much smaller than the number of legal cases in reality. therefore, annotating law articles requires much less effort than annotating legal cases. 2) generaliability: legal knowledge from articles is applicable to all cases under the corresponding legal system. notably, the idea of annotating law articles as fine-grained legal knowledge can be applied to all statutory law systems, or even to the common law systems, because statutory law is one of the most important sources of common law. therefore, the effectiveness of the proposed methodology in this paper is not limited to a specific legal system. experimental results on multiple legal case retrieval datasets demonstrate that caseencoder significantly outperforms the existing general pretraining models and legal-specific pre-training models. we also present case document embedding visualizations to showcase the potential of caseencoder in downstream tasks such as charge prediction and article prediction. ## related work the methods for legal case retrieval can be divided into two categories: traditional bagof-words methods and neural-based methods. before the emergence of deep neural networks, tf-idf+vsm @xcite , bm25 @xcite , and lmir (ponte and croft, 1998) are representative bag-of-words methods. these methods have one common feature: they all treat documents as a sequence of terms and compute the document similarity score by statistical factors like term frequency or inverse document frequency. one limitation of such methods is that they ignore the information of the order among terms. with the rapid development of nlp techniques, applying pre-trained language models (plms) to legal case retrieval has received huge success. @xcite ; @xcite directly retrieve legal cases through a fine-tuned bert @xcite . @xcite and @xcite use large chinese legal corpus to pre-train bert and longformer, respectively. in addition to applying original plms to the legal domain, there are also works trying to incorporate legal knowledge into their proposed models. for example, @xcite propose law-match, a model-agnostic causal learning method that utilizes the knowledge in law articles to tackle legal case matching. @xcite integrate the retrieval, processing, and information extraction into a pipeline for the recognition of key information in refugee cases. their model retrieves a total of 19 categories of items from refugee cases before conducting information extraction by the state-of-the-art neural named-entity recognition method. bhattacharya title: xxx's case of hazardous driving court: people's court of xxx zone, shandong province case number: criminal judgment (2018) lu (xxxx) criminal no. xxx parties' information public prosecutor: shandong province xxx people's procuratorate defendant background: defendant xxx, male, was born in xxx… fact: (accusation of the prosecutor) on december 18, 2017, the defendant xxx was drunk and drove an ordinary two-wheeled motorcycle along fushun road in weihai city when it collided with a small car… (evidence) …and provided the following: 1. the arrest issued by the public security bureau of xxx; 2. testimony of witness xxx … (facts confirmed by the court) after investigation, it was found that on december 18, 2017, the defendant xxx was drunk and drove … holding: the court believes that the defendant xxx violated the transportation management regulations and drove a motor vehicle on the road drunk, and his behavior constituted the crime of hazardous driving… decision: the defendant xxx is guilty of hazardous driving, sentenced to one month and fifteen days of detention… judge xxx march 30, 2018 clerk xxx et al. (2022) incorporate domain knowledge-based document similarity into pcnet (minocha et al., 2015) and propose a heterogeneous network called hier-spcnet for a better representation of document similarity. however, these neural-based works all require task-specific fine-tuning or alignment to reach relatively good performance. by comparison, our proposed method can be directly applied to different downstream tasks in the zeroshot setting. ## task definition given a query case q, the task aims to retrieve relevant cases from a candidate list @xmath0, c 2 , ..., c m }, where m is the size of l, and rank them by the relevance to q. each candidate case document in the list has three main components: 1. facts are objective fact statements confirmed by the court based on the evidence provided by the defendant and plaintiff. these statements typically answer questions such as where, when, and how the case occurred. 2. holding is the judge's opinion on the key arguments of the case. it explains the reasoning behind the judge's decision. 3. decision contains the final judgment of the defendant including the charge, sentence, and articles involved. this component is the official outcome of a case. an example of a legal case document is shown in figure 1 . in the real legal case retrieval scenario, q is usually a case to be judged. therefore, q only contains the facts of a case, while candidate cases are precedents with a complete document including a title, meta information, facts, holding, decision, and related law articles. in this paper, we focus primarily on retrieving cases under criminal law. to match the real legal scenario that the retrieval model might have never seen q or its related cases before, in this paper, all models are evaluated in a zero-shot manner. ## method this section outlines the design and implementation of caseencoder. figure 2 illustrates the overall framework of the model. we begin by introducing the fine-grained case sampling method used for data preparation in section 4.1. then, in section 4.2, we describe the pre-training tasks proposed in caseencoder. ## conclusion and limitation this paper proposes caseencoder, a pre-trained encoder that utilizes fine-grained legal knowledge to enhance the representation of case document embeddings. by introducing law article annotation into the sampling method, we improve the quality of sampled positive and negative cases during data preparation. next, in the pre-training stage, caseencoder adopts two legal-specific pre-training tasks to align with the relevance judgment criteria in the legal domain. experiments and visual analysis demonstrate the effectiveness of case embeddings generated by caseencoder in solving zeroshot legal case retrieval. the main limitations of caseencoder are: firstly, the definition of similar legal cases is based on the assumption that "cases committing the same crime have articles in common", which is not applicable to all legal systems. however, the idea of annotating law articles as fine-grained legal knowledge to enhance the performance of plms has the potential to be adopted in other law systems. in the future, we will consider validating the effectiveness of caseencoder on datasets of different law systems and languages. we will also explore the potential of caseencoder for different downstream legal tasks. secondly, the transitive rule in section 4.2.2 is not consistent under all circumstances. for example, suppose case a and b are in the same class, and b and c are in the same class, then a and c still might have no law articles in common if case a and c both belong to multiple classes. according to our experimental results, the influence of this special case can be alleviated by reducing the batch size in the pre-training stage, because the possibility of two multiple-class cases appearing in the same training batch drops with the reduction of batch size. we will leave the optimization of our proposed bcl function as our future work.