Prototype-Based Interpretability for Legal Citation Prediction (2023)

Abstract: Deep learning has made significant progress in the past decade, and demonstrates potential to solve problems with extensive social impact. In high-stakes decision making areas such as law, experts often require interpretability for automatic systems to be utilized in practical settings. In this work, we attempt to address these requirements applied to the important problem of legal citation prediction (LCP). We design the task with parallels to the thought-process of lawyers, i.e., with reference to both precedents and legislative provisions. After initial experimental results, we refine the target citation predictions with the feedback of legal experts. Additionally, we introduce a prototype architecture to add interpretability, achieving strong performance while adhering to decision parameters used by lawyers. Our study builds on and leverages the state-of-the-art language processing models for law, while addressing vital considerations for high-stakes tasks with practical societal impact.

## introduction deep learning has made significant progress in the past decade. researchers have begun applying state-of-the-art methods to problems with extensive social impact, which in turn brings about many critical challenges for these deep learning models. in high-stakes problems, it is essential to understand whether models follow the same reasoning as domain experts or practitioners, and if not, why these models arrive at their final outcomes @xcite . decisions, in other words, often require human comprehensibility for experts to validate and trust their correctness @xcite . the field of law is one such high-stakes domain where decision making processes often have a significant societal impact. in this paper, we study a key problem, citation prediction, which aims to predict the most fitting legal citation for a text passage. legal citation @xcite ) is a fundamental component of a lawyer's argument construction process. unlike in other domains, citations in law are not just used to reference previous works. they serve to indicate "the nature of the authority upon which a statement is based" @xcite . court rules go so far as to authorize judges to reject arguments that are not supported by cited authority, and lawyers who appeal on the basis of arguments for which they have cited no authority can be sanctioned @xcite . for this reason, any system built to address this high-stakes task must also present evidence to properly support the quality of legal argumentation. existing works on legal citation-based tasks @xcite do not adhere to the thoughtprocess employed by lawyers, leading to discrepancies or overlap with existing tasks such as legal judgment prediction. specifically, existing works primarily predict legal citations solely from the facts of a case. while facts are important to understand a situation, lawyers often develop a legal argument by translating facts in an abstract legal problem first. in other words, citations draw from legal reference to strengthen the lawyer's position in court. many citations are not made to ascertain the nature of the violation. they are instead decided by an understanding of the legal issue the lawyer wants to address, as well as how it relates to prior literature and provisions available in the law. in this research, we propose a new definition for this task called legal citation prediction (lcp). this new approach for citation prediction, illustrated in figure 1 and described in section 3.1, aims to mimic a lawyer's reasoning by providing prior literature, hereby referred to as precedents, and provisions of legislature, provisions, as input. to address the critical issue of interpretability, we extend a prototype-based architecture to better serve the requirements of this task @xcite . utilizing prototypes allows the model to formulate representative examples of each citation and draw similarities to legal references. to the best of our knowledge, this work represents the first attempt to use prototype-based interpretability for language model tuning towards a high-stakes legal task. the main contributions of this work include: ## prototype-based legal citation prediction we propose a prototype-based architecture to address the task of legal citation prediction. the full task is described in section 3.1, and our architecture is shown in figure 2 . prototype-based architectures add interpretability by basing their predictions on similarity to "prototypes", which are representative training samples for each target label. our proposed architecture enhances interpretability and grounds the model decision in a lawyer's thought process via a customized loss function that considers precedents and provisions as comparison points. for our lcp task, we use a combination of automatically discovered prototypes for precedents and manually chosen prototypes for provisions. we use vanilla fine-tuning as a baseline, where we append a linear classification head to a pre-trained language model and fine-tune all parameters with multi-class cross-entropy loss, only using the case text as input. at this stage, we also explore performance tradeoffs of task configurations such as limiting text spans and the number of target labels. we also incorporate expert feedback from initial results to adjust the experiment parameters. then, we fine-tune from the base pre-trained model while adding our prototype-based custom loss objective, encouraging an interpretable latent space organized around precedents and provisions. we explore performance trade-offs and the learned embedding space through perturbations. ## results and analyses baseline results of our baseline experiments with vanilla fine-tuning are shown in we add a 45-label setting with only non-procedural citations based on expert feedback in section 4.2. under the 20-label setting, we observe that inputs without context filtering and naively taking the beginning of a document results in the worst performance. providing context of 4 sentences preceding and following the required citation, denoted @xmath0, leads to better results. finally, by @xmath1, we obtain the best performance. as expected, we observe lower f1 scores for 100 labels due to the increase in rare classes. for subsequent experiments, we maintain 20 labels as the baseline for the following reasons: 1) we observe strong empirical evidence for this setting, and 2) the number of labels corresponds to a reasonable number of outcomes which can correspond to prior filtering by legal professionals, and our system would help with disambiguation. the performance of 45 labels without context fol-lows a similar trend, and performance falls between 20 labels and 100 labels. however, once we provide surrounding context to the model, we see a significant increase in performance, @xmath2. compared to the other settings that exhibit 15-22 points in f1 improvement, it is clear that context is more important with legally relevant citations, or procedural citations are more likely to be mentioned in the first 512 tokens. prototype-based model next, we test the prototype-based model based on the best performing baseline configuration @xmath3, summarized in over these experiments, the prototype-based loss results in comparable performance to vanilla fine-tuning while only tuning on the d p reced term. however, the 20-label task (75% procedural citations) sees dramatic improvements from the d p rovis term, outperforming vanilla fine-tuning by 4 points in macro-f1. conversely, the d p rovis term with the same hyperparameters seems to hinder the performance of the 45-label task, decreasing by 4 points in macro-f1. perturbations to further validate the feature importance of the provisions, we perform several perturbations from our p reced + p rovis model. we attempt two settings: 1) keyword masking, i.e. replacing the input keywords with the [mask] token, and 2) random masking, where we randomly mask 15% of the input tokens. for keyword masking, we use a statistical keyword extractor, yake @xcite , to extract 20 ngrams, up to @xmath4, from the provision text. all results are summarized in vanilla fine-tuning vs. prototypes additionally, we compare vanilla fine-tuning to prototypebased training by freezing the latent space, as denoted by the freezing encoder experiment in we visually inspect the learned information by projecting the latent space of the 20-label models. we reduce the dimensionality of the embeddings with umap @xcite to produce 2d projections shown in figure 4 . comparing the baseline to our prototype-based loss, the latter is visually more well-organized; there are fewer outliers at the edges of the latent space and clear clusters of cases corresponding to different citations. after further investigation into the architecture, we observed there was only one prototype discovered for each target citation after vanilla fine-tuning. the remaining cluster centroids did not meet the cosine similarity threshold. this likely increases redundancies in the activations, which makes it easier to find patterns in embedding distances. however, the prototypes are arbitrarily located in the latent space, so the model might be learning spurious prototype activation patterns. ## conclusion we study legal citation prediction (lcp), a problem that serves as a foundational function for decision making in modern legal systems with societal impact. in our work, we built an inherently interpretable prototype architecture that is compatible with any language model encoder, and explains its decisions based on similarity to precedents and provision text of the target citation. we automatically discover prototypes from the training set for each target citation, which reduces the need for expert input, and make predictions based on an input's similarity to these prototypes. through empirical study, we show strong evidence of our architecture towards lcp, offering more interpretability than vanilla fine-tuning for equivalent performance. we also demonstrate that leveraging a combination of precedents and the target provisions' texts in the during model training results in comparable or greater performance to the baseline language model, although the effectiveness of our full architecture depends on the legal nature of the target citations. compared to vanilla fine-tuning, our model's latent space visibly separates different citation embeddings into distinct groups. it is possible to apply our distance-based classification head to an encoder trained with vanilla fine-tuning, but the classification is likely based on spurious features. in practice, this system could be extended to any citation target that has prior examples and source text, such as prior cases (i.e. caselaw). interpretability is a crucial requirement for deploying transparent ai systems, and we encourage more work in this direction for applications with significant social impact.