ILDC for CJPE : I ndian Legal Documents Corpus for Court Judgment Prediction and Explanation (2021)

Abstract: An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a system to be practically useful, predictions by the system should be explainable. To promote research in developing such a system, we introduce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court decisions. A portion of the corpus (a separate test set) is annotated with gold standard explanations by legal experts. Based on ILDC, we propose the task of Court Judgment Prediction and Explanation (CJPE). The task requires an automated system to predict an explainable outcome of a case. We experiment with a battery of baseline models for case predictions and propose a hierarchical occlusion based model for explainability. Our best prediction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed algorithm reveals a significant difference in the point of view of the algorithm and legal experts for explaining the judgments, pointing towards scope for future research.

## introduction in many of the highly populated countries like india, there is a vast number of pending backlog of legal cases that impede the judicial process @xcite . the backlog is due to multiple factors, including the unavailability of competent judges. therefore, a system capable of assisting a judge by suggesting the outcome of an ongoing court case is likely to be useful for expediting the judicial process. however, an automated decision system is not tenable in law unless it is well explained in terms of how humans understand the legal process. hence, it is necessary to explain the suggestion. in other words, we would like such a system to predict not only what should be the final decision of a court case but also how one arrives at that decision. in this paper, we introduce indian legal doc-uments corpus (ildc) intending to promote research in developing a system that could assist in legal case judgment prediction in an explainable way. ildc is a corpus of case proceedings from the supreme court of india (sci) that are annotated with original court decisions. a portion of ildc (i.e., a separate test set) is additionally annotated with gold standard judgment decision explanations by legal experts to evaluate how well the judgment prediction algorithms explain themselves. based on ildc, we propose a new task: court judgment prediction and @xcite . this task aims to predict the final decision given all the facts and arguments of the case and provide an explanation for the predicted decision. the decision can be either allowed, which indicates ruling in favor of the appellant/petitioner, or dismissed, which indicates a ruling in favor of the respondent. the explanations in the cjpe task refer to sentences/phrases in the case description that best justify the final decision. since, we are addressing mainly the sci cases, one might argue that the usefulness of the task may be limited since, the legislative provisions can always change with time. however, the legal principles of how to apply a given law to a given set of facts remain constant for prolonged periods. judgment prediction and explanation in the cjpe task are far more challenging than a standard text-classification task for multiple reasons. firstly, the legal court case documents (especially in indian context) are unstructured and are usually quite long, verbose, and noisy. there is no easy way of extracting and directly using the facts and arguments. secondly, the domain-specific lexicon used in court cases makes models pre-trained on generally available texts ineffective on such documents. consequently, the standard models need to be adapted to the legal domain for the proposed judgment prediction on court cases. thirdly, explaining prediction in legal documents is considerably more challenging as it requires understanding the facts, following the arguments and applying legal rules, and principles to arrive at the final decision. our main contributions can be summarized as: 1. we create a new corpus, indian legal doc-uments corpus (ildc), annotated with court decisions. a portion of the corpus (i.e. a separate test set) is additionally annotated with explanations corresponding to the court decisions. we perform detailed case studies on the corpus to understand differences in prediction and explanation annotations by legal experts, indicative of the computational challenges of modeling the data. 2. we introduce a new task, court judgment prediction and explanation (cjpe), with the two sub-tasks: (a) court judgment prediction (cjp) and (b) explanation of the prediction. while cjp is not a novel task per se; however, in combination with the explanation part, the cjpe task is new. moreover, the requirement for explanations also puts restrictions on the type of techniques that could be tried for cjp. in the cjpe task, gold explanations are not provided in the train set; the task expects that the trained algorithms should explain the predictions without requiring additional information in the form of annotations during training. 3. we develop a battery of baseline models for the cjpe task. we perform extensive experimentation with state-of-the-art machine learning algorithms for the judgment prediction task. we develop a new method for explaining machine predictions since none of the existing methods could be readily applied in our setting. we compare model explainability results with annotations by legal experts, showing significant differences between the point of view of algorithms and experts. ildc is introduced to promote the development of a system/models that will augment humans and not replace them. we have covered the ethical considerations in the paper. nevertheless, the com-munity needs to pursue more research in this regard to fully understand the unforeseen social implications of such models. this paper takes initial steps by introducing the corpus and baseline models to the community. moreover, we plan to continue to grow, revise and upgrade ildc. we release the ildc and code for the prediction and explanation models via github foot_0 . ## related work there has been extensive research on legal domain text, and various corpora and tasks have been proposed e.g., prior case retrieval @xcite , summarization @xcite , catchphrase extraction @xcite , crime classification @xcite , and judgment prediction @xcite . ## indian legal document corpus in this paper, we introduce the indian legal documents corpus (ildc), a collection of case proceedings (in the english language) from the supreme court of india (sci). for a case filed at the sci, a decision ("accepted" v/s "rejected") is taken between the appellant/petitioner versus the respondent by a judge while taking into account the facts of the case, ruling by lower court(s), if any, arguments, statutes, and precedents. for every case filed in the supreme court of india (sci), the judge (or a bench) decides on whether the claim(s) filed by the appellant/petitioner against the respondent should be "accepted" or "rejected". the decision is relative to the appellant. in ildc, each of the case proceeding document is labeled with the original decision made by the judge(s) of the sci, which serve as the gold labels. in addition to the ground truth decision, a separate test set documents are annotated (by legal experts) with explanations that led to the decision. the explanations annotations are ranked in the order of importance. ## annotation analysis we performed a detailed analysis of case predictions and the explanations annotations. with assistance from a legal expert, we also performed detailed studies for some court cases to understand the task's complexity and possible reasons for deviations between the annotators. ## cjpe task given a case proceeding from the sci, the task of court judgment prediction and expla-nation (cjpe) is to automatically predict the decision for the case (with respect to the appellant) and provide the explanation for the decision. we address the cjpe task via two sub-tasks in the following sequence: prediction and explanation. prediction: given a case proceeding d, the task is to predict the decision @xmath0, 1}, where the label 1 corresponds to the acceptance of the appeal/petition of the appellant/petitioner. explanation: given the case proceeding and the predicted decision for the case, the task is to explain the decision by predicting important sentences that lead to the decision. annotated explanations are not provided during training; the rationale is that a model learned for prediction should explain the decision without explicit training on explanations, since explanation annotations are difficult to obtain. ## conclusion this paper introduces the ildc corpus and corresponding cjpe task. the corpus is annotated with case decisions and explanations for the decisions for a separate test set. analysis of the corpus and modeling results shows the complexity of legal documents that pose challenges from a computational perspective. we hope that the corpus and the task would provide a challenging and interesting resource for the legal nlp researchers. for future work, we would like to train a legal transformer similar to legal-bert @xcite on our indian legal case documents. moreover, we would also like to focus upon using rhetorical roles @xcite of the sentences to include structural information of the documents for cjpe task as well.