Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models (2024)

Abstract: This study investigates judgment prediction in a realistic scenario within the context of Indian judgments, utilizing a range of transformer-based models, including InLegalBERT, BERT, and XLNet, alongside LLMs such as Llama-2 and GPT-3.5 Turbo. In this realistic scenario, we simulate how judgments are predicted at the point when a case is presented for a decision in court, using only the information available at that time, such as the facts of the case, statutes, precedents, and arguments. This approach mimics real-world conditions, where decisions must be made without the benefit of hindsight, unlike retrospective analyses often found in previous studies. For transformer models, we experiment with hierarchical transformers and the summarization of judgment facts to optimize input for these models. Our experiments with LLMs reveal that GPT-3.5 Turbo excels in realistic scenarios, demonstrating robust performance in judgment prediction. Furthermore, incorporating additional legal information, such as statutes and precedents, significantly improves the outcome of the prediction task. The LLMs also provide explanations for their predictions. To evaluate the quality of these predictions and explanations, we introduce two human evaluation metrics: Clarity and Linking. Our findings from both automatic and human evaluations indicate that, despite advancements in LLMs, they are yet to achieve expert-level performance in judgment prediction and explanation tasks.

## introduction predicting case outcomes based on judgesummarized narratives is an important task. unlike previous studies @xcite and @xcite , we aim to simulate realistic scenarios where legal judgment prediction systems are used to predict and explain judgments as cases arrive on the bench for adjudication. our approach focuses on the core factual components of the case-specifically, the events that led to the case being filed, which serve as the basis for judgment prediction. these facts are the foundation of legal arguments and provide the context needed for making judicial decisions. in contrast to previous works that have included the entire case text (including proceedings), our focus on facts mirrors real-world conditions, where judges rely primarily on the case facts when delivering judgments. in addition to the facts of the case, we incorporate additional legal information such as statutes, precedents, and arguments. statutes represent codified legal principles, while precedents provide casespecific rulings that help guide decision-making. together, these legal frameworks offer a structured basis upon which judges rely when formulating their rulings. by extracting and integrating these elements into our models, we aim to enhance both the prediction and explanation tasks by grounding the analysis in actual legal texts and the governing principles that are applied in real cases. we explore the efficacy of various transformerbased models investigate the impact of summarizing legal judgments @xcite @xcite using techniques @xcite @xcite such as bert-sum @xcite , casesummarizer @xcite , letsum @xcite , and summarun-ner @xcite . our findings suggest that leveraging summarized information yields decent results in judgment prediction. to further enhance the quality of prediction, we introduce hierarchical transformer models that utilize the entirety of judgment facts, demonstrating superior performance compared to traditional summarization methods. additionally, our examination of llms, including llama-2 (13b & 70b) @xcite and gpt-3.5 turbo (brown, 2020), highlights the exceptional performance of gpt-3.5 turbo in the context of indian legal judgment pre-diction. we find that augmenting our models with additional legal information, such as statutes, precedents, and arguments, significantly improves the quality of both tasks. in addition to focusing on the accuracy of legal judgment prediction, it is equally important to assess the quality of the explanations provided by the models. for this reason, we introduce two novel human evaluation metrics: clarity and linking. clarity refers to how well the predictions and explanations are structured and whether they convey the reasoning in a clear and understandable manner. this is critical in the legal domain, where complex legal concepts must be communicated effectively. linking, on the other hand, evaluates the logical consistency between the explanation and the final judgment. it assesses whether the explanation effectively ties back to the outcome and supports the predicted decision. these metrics are vital because, while models may produce accurate predictions, their explanations often lack coherence or fail to justify the decision meaningfully. by incorporating these metrics, we aim to ensure that models provide not only accurate outcomes but also transparent and interpretable explanations that can be trusted by legal professionals. the key contributions of this study are: to ensure reproducibility, both the code and dataset have been made publicly available via our repository 1 . additionally, for convenience, we have uploaded the data 2 and models 3 to huggingface. ## related work the field of legal judgment prediction (ljp) has seen significant advancements, driven by the need to automate legal case outcome forecasting and alleviate the burden of overwhelming caseloads. early works by @xcite , @xcite , and @xcite laid the foundation for ljp, emphasizing the importance of explainability in ai predictions. benchmark datasets such as cail2018 @xcite , echr-cases @xcite , and others have spurred research in this area, inspiring models like topjudge and mlcp-nln. however, there remains a gap between machine and human performance. in the indian context, datasets like ildc @xcite , predex @xcite and @xcite @xcite have highlighted the growing role of ai in legal judgments, with an emphasis on explainability. research in ljp with llms, such as @xcite and @xcite , has experimented with models like gpt-3.5 turbo and llama-2 on indian legal datasets. other studies, such as @xcite on romanian legal texts and @xcite on korean legal language, have demonstrated ljp's adaptability across legal systems. cross-jurisdictional work, including @xcite , showcases ljp's applicability in different legal frameworks, with research expanding to multilingual considerations, as seen in @xcite and @xcite for hindi legal documents. recent innovations, such as event extraction and multi-stage learning @xcite , continue to push the boundaries of ljp research. ## task definition this study focuses on supreme court of india (sci) judgments, and the court judgment prediction with explanation task consists of two subtasks: task a: judgment prediction: this subtask is framed as a binary classification problem specific to sci cases. given a segment of the legal judgment as input, the goal is to predict whether the decision favors or is against the appellant. the prediction is represented by binary labels: {1, 0}, where 1 indicates that the appeal is accepted (i.e., if any part of the appeal is accepted, the decision is considered in favor of the appellant). although some cases might involve multiple heads of appeal, where an appellant might win on some grounds and lose on others, for the purposes of this task, the outcome is simplified to a binary decision. cases with mixed outcomes are excluded or reduced to this binary format for prediction. task b: rationale explanation: this subtask involves generating a coherent explanation or rationale that justifies the predicted decision, based on the provided segment of the judgment. the explanation seeks to clarify the reasoning behind the predicted outcome. the workflow of the system, as illustrated in figure 1 , captures the entire process-from extracting facts and additional legal information (such as statutes, precedents, and lower court rulings) to feeding this data into transformer models, hierarchical transformers, and llms. the diagram visually represents the pipeline of both tasks, highlighting how the prediction and explanation processes interact to form a comprehensive legal judgment prediction system. ## dataset we utilize the ildc-multi dataset, as described by @xcite , which comprises a total of 34,816 legal judgments from the supreme court of india, collected from 1947 to april 2020 via the indian kanoon website 4 . this dataset is divided into three subsets: training, validation, and test which contains 32,305, 994, and 1,517 judgments correspondingly. it is specifically designed to support the tasks of court judgment prediction and explanation (cjpe), with a portion of the legal judgment serving as input for both prediction and explanation processes. additionally, a subset of this corpus is annotated with gold-standard explanations provided by legal experts, enhancing its utility for developing automated systems that predict and explain judicial outcomes. ## conclusions in this study, we explored the effectiveness of various llms and transformer architectures in the task of judgment prediction and explanation using the ildc-multi dataset. our results demonstrate that incorporating additional case-specific information significantly enhances the prediction accuracy compared to using only factual information. the results also highlight the superiority of hierarchical transformer models over traditional summarization techniques, suggesting that a comprehensive approach to input data yields better predictive outcomes. despite the promising results, our evaluations reveal that automated metrics still fall short of matching the performance levels of human legal experts, who demonstrate a high degree of agreement in judgment assessments. this gap underscores the need for further refinement of llms and transformer models to improve their interpretability and reliability in legal contexts.