LJPC heck: Functional Tests for Legal Judgment Prediction (2024)

Abstract: Legal Judgment Prediction (LJP) refers to the task of automatically predicting judgment results (e.g., charges, law articles and term of penalty) given the fact description of cases. While SOTA models have achieved high accuracy and F1 scores on public datasets, existing datasets fail to evaluate specific aspects of these models (e.g., legal fairness, which significantly impact their applications in real scenarios). Inspired by functional testing in software engineering, we introduce LJPCHECK, a suite of functional tests for LJP models, to comprehend LJP models’ behaviors and offer diagnostic insights. We illustrate the utility of LJPCHECK on five SOTA LJP models. Extensive experiments reveal vulnerabilities in these models, prompting an in-depth discussion into the underlying reasons of their shortcomings.

## introduction legal judgment prediction (ljp) aims to automatically predict judgment results given the fact description of a case. while english ljp focuses on violation prediction @xcite @xcite , chinese ljp focuses on charge prediction, law article prediction and term of penalty prediction @xcite . there also exists ljp work in other languages, such as french @xcite and greman @xcite . figure 1 illustrates an example of the three subtasks involved in chinese ljp. such ljp systems can assist legal professionals in case processing as well as offering low-cost consulting services to nonprofessionals. how should the quality of ljp models be evaluated? currently ljp models are evaluated using generic metrics such as f1 and accuracy on specific ljp subtasks such as charge prediction. however, a simple one-metric evaluation is far from sufficient for ljp since it does not provide an in-depth understanding of how a given model behaves, its weaknesses, and how to improve its performance in general. for instance, judicial fairness, which highlights that the parties involved in a case should be judged based on the case facts rather than demographic information such as gender and age, is one of the crucial considerations in the legal domain that cannot be evaluated using generic metrics. the reason is that a model that achieves a poor accuracy on, for instance, charge prediction can be fair as long as it does not make predictions based on race and gender, whereas a model that performs well on charge prediction can be unfair if it exploits race or gender information during its prediction process. while the use of one-metric evaluation is not uncommon in nlp and the shortcomings of such kind of evaluations are applicable to all nlp tasks, what is characteristic about ljp is its humancentered nature: understanding why a particular ai tool decided to charge someone with imprisonment is crucial for any real-world applications. motivated in part by the above observation, we propose a new approach to the analysis of ljp models that allows for targeted testing of specific aspects of ljp models, such as fairness. our approach is inspired by functional testing in soft-ware engineering @xcite , which is a testing framework that assesses different functionalities of a given model by validating its output on sets of targeted test cases. specifically, we introduce ljpcheck, a suite of functional tests for ljp models that are sufficiently complex to allow better comprehension of a model's behavior and offer diagnostic insights. the contributions of our work are four-fold: a new approach to the analysis of ljp models. to our knowledge, we are the first to employ functional testing in software engineering to augment the evaluation of ljp models to include specific aspects, such as fairness. an easily extensible methodology. while our test suite is developed for evaluating chinese ljp models (i.e., some of the test cases are developed with specific chinese law articles in mind), the same methodology, which we borrow from @xcite , can naturally be applied to the development of test suites for other jurisdictions. more broadly, this methodology is task-agnostic and can therefore be applied to other nlp tasks. a new test suite. our test suite 1 , which spans over 10,800 examples, provides a valuable resource for future research into the ljp direction. an analysis of sota ljp models. while the vast majority of recent work on ljp has focused on number crunching, where the goal is to improve sota results, we focus on understanding the strengths and weaknesses of sota ljp models. we believe that our work is also valuable for the broader "analysis-focused research" because it shows how to apply the methodology of @xcite to a new domain. ## related work much recent work on ljp has focused on applying deep learning and proposing different kinds of neural networks to solve the subtasks in ljp, with the primary goal of improving performance numbers. for instance, some researchers @xcite extract key elements from facts to improve both accuracy and interpretability, while others employ multi-task learning @xcite to simultaneously predict multiple judgments. with the development of pretrained language models (plms), legal plms like 1 the test suite is available at here (link). lawformer @xcite pretrained on legalrelated data is used to solve different legal tasks (including ljp) in a unified framework. nevertheless, given the significant impact of legal decisions on individuals, the evaluation of ai systems in the legal domain is getting attention @xcite . @xcite train a word2vec model on a us case law dataset and discover that nlp methods make undesirable distinctions between legally equivalent entities that vary only by race, which leads to race bias. @xcite propose a multi-lingual benchmark suite, based on four legal datasets across four jurisdictions, to test model biases in attributes like demographics and court regions. @xcite design a framework to evaluate whether existing charge prediction models are trustworthy with 3 proposed principles that they believe models should follow. @xcite propose a metric to measure fairness in the legal domain, highlighting that both regional and gender biases exist in the legal systems. in this paper, we provide an evaluation framework tailored to ljp systems. unlike the above works which only focus on one or a few aspects, we give broader insights with respect to 23 different functionalities. further, our test framework can be applied to other tasks whereas the above works are tied to their own tasks. ## ljpcheck functional testing in software engineering refers to the practice of testing specific functionalities of software applications by providing sample inputs and verifying whether the outputs are as expected. while prediction accuracy is a key consideration to ljp models, there are other aspects that are essential but overlooked when evaluating models. we define these considerable aspects as functionalities in this paper. for model-agnostic testing, we apply the principle of decoupling testing from implementation by treating models as black boxes, which allows different models to be tested without knowing their internal structures. next, we detail the functionalities to be tested (e.g. time fairness), the test types used (e.g. minimum functionality test), and how we generate tests. ## experiments we test five sota ljp models using ljpcheck. ## conclusion we proposed using functional testing for a more comprehensive evaluation of ljp models. we introduced ljpcheck, which contains 23 functionalities targeted at testing different aspects of ljp systems. five sota ljp models are tested and weaknesses are revealed. in future work, we aim to explore testing model interpretability given its critical importance in ljp. we also plan to incorporate the results of law article prediction task, which were not utilized in our current tests, to enhance the completeness and reliability of our tests.