CMDL : A Large-Scale C hinese Multi-Defendant Legal Judgment Prediction Dataset (2024)

Abstract: Legal Judgment Prediction (LJP) has attracted significant attention in recent years. However, previous studies have primarily focused on cases involving only a single defendant, skipping multi-defendant cases due to complexity and difficulty. To advance research, we introduce CMDL, a large-scale real-world Chinese Multi-Defendant LJP dataset, which consists of over 393,945 cases with nearly 1.2 million defendants in total. For performance evaluation, we propose case-level evaluation metrics dedicated for the multi-defendant scenario. Experimental results on CMDL show existing SOTA approaches demonstrate weakness when applied to cases involving multiple defendants. We highlight several challenges that require attention and resolution.

## introduction legal judgment prediction (ljp) aims to predict judgment outcomes (e.g., law articles, charges) given the fact description of a case. ljp has been widely studied in various jurisdictions and languages @xcite @xcite @xcite . english ljp @xcite focuses on violation prediction while swiss ljp focuses on predicting plaintiff's claims @xcite . chinese ljp primarily focuses on predicting charges, applicable law articles and terms of penalty in criminal cases. ljp can help both legal professionals in analyzing cases and nonprofessionals with low-cost consulting services. in this paper, we focus on chinese ljp. a variety of methods have been proposed to solve chinese ljp. some improve performance by exploring the relationships between subtasks and employ multitask learning @xcite @xcite . some extract key information from facts for prediction, enhancing both accuracy and interpretability @xcite . the importance of distinguishing between similar charges and legal provisions has also attracted attention @xcite . with the development of pre-trained models, researchers begin to propose ljp solutions based on these models and achieve new state-of-the-art results @xcite . nevertheless, existing methods are mostly designed for and tested on single-defendant cases, leaving out multi-defendant cases due to complexity @xcite . while many researchers highlight the significance of exploring multi-defendant cases in future studies @xcite , it remains a relatively underexplored area with limited satisfactory solutions. given the fact description of a case, multidefendant ljp aims to predict judgment outcomes for each defendant. it's nontrivial to solve multidefendant ljp. as shown in figure 1 , three defendants involved in a single case receive different judgments as they face different sentencing factors based on judicial theory @xcite . for instance, defendant wang involves in the case as the role of accomplice that does not directly participate in the theft activity, whereas defendant liu and xu are principle criminals for stealing. thus, defendant wang gets a lighter punishment compared to the others. however, there are several challenges for models to distinguish sentencing factors (e.g. identifying the roles in joint crimes) among defendants as follows. segmenting facts. legal documents do not follow a sequential pattern where each defendant is individually described in facts. instead, facts are typically presented in chronological order, with details concerning each defendant interspersed and not separated. moreover, extracting details pertinent to each defendant solely by their names is impractical, as it may result in disjointed fragments. injecting relevant legal knowledge. judging cases with multi-defendants necessitates adherence to specific legal logic. accordingly, models should possess the ability to analyze facts and integrate applicable legal knowledge. as shown in figure 1 , while defendant liu and xu are both principal criminals, they receive different penalties due to xu's mitigating circumstance of voluntary surrender. while there are numerous single-defendant ljp approaches, they are not applicable in the multi-defendant setting as they lack the necessary legal knowledge required for accurately predicting multi-defendant cases. to facilitate multi-defendant ljp research, we present cmdl, a large-scale chinese multi-defendant ljp dataset. cmdl consists of 393,945 criminal cases involving multiple defendants, covering 321 charges and 275 law articles. compared with existing multi-defendant ljp datasets @xcite , cmdl contains over ten times the number of cases and the number of labels than others. cmdl also contains more comprehensive legal annotations, including (1) detailed fines (either specific amounts or being fined without a specified amount), (2) diverse term of penalty (e.g., imprisonment, surveillance, etc.) and (3) more fine-grained paragraph-level law articles. for evaluating models' performance under the multi-defendant setting, we propose case-level met-rics, which will be described in 4.2.1. we conduct extensive experiments to uncover the challenges of multi-defendant ljp, and experimental results show multi-defendant ljp is still far from being resolved and requires further investigation. in summary, our contributions are three-fold: a large-scale multi-defendant ljp dataset. to the best of our knowledge, cmdl is the largest chinese multi-defendant ljp dataset covering the widest range of charges, law articles and penalties. it can also be easily expanded and updated with our automatic annotation process. we believe our dataset can generate more interest in multidefendant ljp and promote further research in the field of ljp. a more comprehensive approach to evaluate model performance on multi-defendant cases. given the varying number of defendants in multidefendant cases, we believe it's important to assess model performance not only at the individual defendant level (by taking each defendant as a sample) but also from a holistic perspective of the case. ## related work ljp as an important application of ai in the legal field has been extensively studied. researches in other languages and jurisdictions, especially in countries employing the common law system, typically address binary classification tasks such as predicting violation @xcite or whether the plaintiff's claims will be accepted/rejected @xcite @xcite . to the best of our knowledge, no ljp research from other countries involves multiple defendants. therefore, our discussion here mainly focus on the context of chinese ljp. in recent years, deep learning and neural networks have emerged as the primary methods for solving various ljp subtasks. researchers focus on improving accuracy and interpretability by extracting key elements to stimulate practical judicial process @xcite @xcite , exploring relationships cail2018 multiljp cmdl case defendant single multiple multiple # cases 2,676,075 23,717 393,945 # defendants 2,676,075 80,477 1,199,117 # charges 202 23 321 # law articles 183 22 275 fines ✓ × ✓ paragraph-level law × × ✓ non-imprisonment term × × ✓ ## cmdl in this section, we first describe the definition of multi-defendant ljp task in our dataset. then we elaborate on the dataset's annotation process and present the data analysis. ## experiments in this section, we apply state-of-the-art models to multi-defendant ljp based on cmdl, focusing on evaluating their performance and identifying the underlying challenges. ## conclusion this paper propose the first large-scale real-world chinese multi-defendant dataset for ljp, which includes 393,945 multi-defendant cases, encompassing a total of nearly 1.2 million defendants and 321 charges. cmdl contains comprehensive annotations including fines, diverse terms of penalty, and paragraph-level law article labels. experimental results indicate that multi-defendant ljp is challenging and requires further efforts for improvement.