LLM s – the Good, the Bad or the Indispensable?: A Use Case on Legal Statute Prediction and Legal Judgment Prediction on I ndian Court Cases (2023)

Abstract: The Large Language Models (LLMs) have impacted many real-life tasks. To examine the efficacy of LLMs in a high-stake domain like law, we have applied state-of-the-art LLMs for two popular tasks: Statute Prediction and Judgment Prediction, on Indian Supreme Court cases. We see that while LLMs exhibit excellent predictive performance in Statute Prediction, their performance dips in Judgment Prediction when compared with many standard models. The explanations generated by LLMs (along with prediction) are of moderate to decent quality. We also see evidence of gender and religious bias in the LLM-predicted results. In addition, we present a note from a senior legal expert on the ethical concerns of deploying LLMs in these critical legal tasks.

## introduction the large language models (in short, the llms) have touched upon many ai-based tasks. there has been a recent surge in the application of llms e.g. gpt @xcite , bloom (blo), flan-t5(t5), llama(lla) which has heralded a new area in data science. along with many other areas, llms have been applied to the legal domain. @xcite has applied gpt-3 for statutory reasoning on u.s. statutes with moderate success. similarly, @xcite applied gpt-3 for the legal reasoning task of statute detection on a japanese bar exam dataset effectively. @xcite shows that gpt-4 is able to pass the uniform bar examination (ube), often outperforming humans. however, @xcite reported that chatgpt performs moderately on real exams at the university of minnesota law school. @xcite has designed a multiagent system for judgment prediction using gpt-2 for u.s. supreme court with moderate accuracy. as a different line of work, @xcite leveraged gpt for sentence-level annotation of legal text. recently, @xcite presented a detailed take on the future of law in the light of the large language models. however, the acceptability of llms in a highstake domain like law is still under scrutiny. to this end, we address the following in this paper: (i) we attempt two legal-specific tasks -statute prediction (section 2) and judgment prediction (section 3), using llms in the light of state-of-the-art baselines, on indian supreme court cases. (ii) we generate explanations for each of these tasks using llms and compare them with annotated explanations by legal experts, providing detailed quantitative and qualitative analysis of the same (sections 2 and 3). we also identify errors and hallucinations in the generated explanations (tables 11 and 12 ). (iii) we also discuss the presence of bias in the outputs generated by llms and ethical considerations for llms in law (section 4). (iv) we also release a legal-expert-explanation-annotated statute prediction dataset. to our knowledge, this is among the first works so far on indian cases. in a recent work, @xcite applied llms to legal court case summarization. also, to our knowledge, no prior work has addressed these issues on these legal problems esp. for indian cases. the data and code are available here: https://github.com/ somsubhra04/llm_legal_prompt_generation . ## statute prediction given a fact text @xcite of a case, we consider two types of tasks (i) prediction of statutes (statute predn) and (ii) prediction of statutes along with the generation of explanation for the prediction (statute predn + expln). ## judgement prediction given a whole case (except the final decision), we consider two types of tasks (i) prediction of judgment, viz. allowed (the appellant won the case; label 1) or dismissed (the appellant lost the case; label 0) (we call this task judgment predn) and (ii) prediction of judgment along with the gener- ## conclusion in this paper, we applied state-of-the-art llms for statute prediction and judgment prediction. we see that llms (i) outperform standard baselines in statute prediction, (ii) underperform similar baselines for judgment prediction, (iii) for a subset of cases, generate explanations moderate for statute prediction, but often quite good for judgment prediction and (iv) exhibits evidence of gender and religion bias in the predictions and generated explanations. the commendable performance in a complex task like multi-label statute prediction (also with explanations) with no explicit training in the legal domain, possibly shows promise in the application of llms in future legal tasks. at the same time, judicious monitoring of bias, fairness and ethics in llms is a desideratum.