PILOT : Legal Case Outcome Prediction with Case Law (2024)

Abstract: Machine learning shows promise in predicting the outcome of legal cases, but most research has concentrated on civil law cases rather than case law systems. We identified two unique challenges in making legal case outcome predictions with case law. First, it is crucial to identify relevant precedent cases that serve as fundamental evidence for judges during decision-making. Second, it is necessary to consider the evolution of legal principles over time, as early cases may adhere to different legal contexts. In this paper, we proposed a new framework named PILOT (PredictIng Legal case OuTcome) for case outcome prediction. It comprises two modules for relevant case retrieval and temporal pattern handling, respectively. To benchmark the performance of existing legal case outcome prediction models, we curated a dataset from a large-scale case law database. We demonstrate the importance of accurately identifying precedent cases and mitigating the temporal shift when making predictions for case law, as our method shows a significant improvement over the prior methods that focus on civil law case outcome predictions.

## introduction predicting legal case outcomes is a crucial task that facilitates data-driven decision-making in legal cases based on relevant information, such as the factual description @xcite . with a significant number of legal cases arising worldwide each year, legal professionals face the daunting task of reviewing the extensive legal text and delivering accurate and fair outcomes. legal case outcome prediction has the potential to simplify this labor-intensive document review process, enhancing strategy and decision-making. as the volume and complexity of cases continue to escalate, the development of precise and reliable legal case outcome prediction systems becomes an urgent priority. two legal frameworks exist across the globe: the civil law system, which assesses each case based on comprehensive codes and statutes, and the case law system, where the interpretation and application of law heavily depend on precedent court decisions. most existing works were proposed for the civil law framework, including charge prediction, violated articles prediction, prison terms prediction, court decision prediction, and court view generation @xcite @xcite @xcite . however, predicting case outcomes in case law systems presents unique challenges distinct from those in civil law: (1) it requires the identification of similar historical cases, and (2) meanwhile accounting for the evolution of legal principles over time. to fill the gap, we proposed a new model named pilot (predicting legal case outcome) for case outcome prediction, which consists of two functional modules: to facilitate this line of research, we established a new dataset named echr2023, which was extracted from the european court of human rights (echr) database 1 with focusing on precedent cases and temporal concept shift. we evaluated the proposed pilot model against state-of-the-art models on echr2023. the experiment results show that pilot substantially outperforms existing works in several metrics. the two modules in pilot effectively improve the performance in different aspects. in summary, the main contributions of this paper are as follows • we highlight the issue of temporal pattern shift in legal ai tasks. this problem is important but is usually ignored in most previous works. ## related work legal case outcome prediction on civil law framework has been well studied, mainly focusing on predicting whether the case description violates existing legislation. machine learning technologies, including multi-task learning @xcite , few-show learning @xcite has been adopted. model explanation has been another focused @xcite @xcite @xcite . in contrast, for case law systems that heavily relies on judicial decisions of relevant precedent cases rather than solely on constitutional law when rendering final case outcomes, there are relatively few studies due to the lack of scarcity 1 https://hudoc.echr.coe.int of large-scale, high-quality, and structured labeled data. for instance, @xcite utilize hier-bert to first encode individual facts and then employ two layers of transformers to encode all the facts within a given case. @xcite generate rationales through a text encoder sub-network that reads the text, a rationale extraction sub-network that identifies the most important words via a binary mask, and a prediction sub-network that classifies a hard-masked version of the text. they also incorporate rationale constraints as regularizers. @xcite employ a fact encoding layer to encode facts and a charge encoding layer to encode charges. subsequently, they use a matching layer, which incorporates an attention mechanism, to predict the final charges for each case. @xcite utilize a hierarchical xlnet architecture to predict case outcomes and related interpretations. these efforts primarily focus on the classification of fundamental case outcomes. to the best of our knowledge, most of the existing works do not handle temporal pattern shift. temporal pattern shift arises due to changes in label distribution, meaning, and etc. existing research approaches this issue from different angles. for example, @xcite analyzes the impact of temporal pattern shift on model explanations. @xcite explored drift adaptation through transfer-based ensemble learning. fan et al. @xcite proposed to use two conet networks to model the normalized parameters of historical and future windows separately, enabling normalization and prediction of future sequences. @xcite introduced an out-of-domain representation learning approach utilizing adversarial learning to capture domain-specific segments and a domain-independent commonality representation. @xcite introduced temporal attention and trained a transformer-based model with additional time-based inputs. in the legal field, @xcite tackled temporal pattern shift in legal text classification by proposing label-wise distributional robust optimization. this algorithm addresses temporal pattern shift stemming from class imbalance problems and enhances model robustness. however, the existing works are too general and are designed for simple scenes, so they do not perform well in adapting the more complex shift. there is still a lack of a comprehensive solution for legal models to adapt the shift in the legal field directly and naturally. ## echr202dataset we build a novel dataset called echr2023 that takes the special challenges in legal case outcome prediction with case law. this dataset is derived from the most recent echr database. the primary focus of echr2023 is to investigate the issue of temporal pattern shifts in the legal domain. ## experiments in this section, we conducted extensive experiments to show the performance of pilot associated with more in-depth analysis. universally, we report the average results of all models obtained by five runs with different random seeds, to ensure fair comparison. we use four metrics to evaluate the legal case outcomes: micro-f1, micro-jaccard, micro-pr-auc, and micro-roc-auc. more training details can be found in appendix a. as for the availability of cases during training and evaluation, we strictly ensure that we do not use any later cases as references for the current case. during the training phase, all prior cases from the training set are available as precedents. at test time, all prior cases from both the training and test sets are available. in the contrastive learning of the case encoding model, we only use data from the training split of the dataset. ## conclusion in conclusion, this paper introduces the pilot model to tackle the challenges associated with predicting case outcomes in case law systems. through our experiments, we have demonstrated the superior accuracy of our model in predicting case outcomes compared to existing methods. this improvement can be attributed to the identification of similar cases and the effective handling of temporal pattern changes. moreover, our proposed model goes beyond enhancing the accuracy of legal case predictions. it also offers valuable insights into legal reasoning and the evolution of legal principles. precedent cases hold significant importance within the case law legal framework. it is worth noting that many previous works have primarily focused on the civil law system, which differs from the case law system. by analyzing and leveraging precedent cases, our model provides a deeper understanding of the underlying legal principles and their application.