Enabling Discriminative Reasoning in LLM s for Legal Judgment Prediction (2024)

Abstract: Legal judgment prediction is essential for enhancing judicial efficiency. In this work, we identify that existing large language models (LLMs) underperform in this domain due to challenges in understanding case complexities and distinguishing between similar charges. To adapt LLMs for effective legal judgment prediction, we introduce the Ask-Discriminate-Predict (ADAPT) reasoning framework inspired by human judicial reasoning. ADAPT involves decomposing case facts, discriminating among potential charges, and predicting the final judgment. We further enhance LLMs through fine-tuning with multi-task synthetic trajectories to improve legal judgment prediction accuracy and efficiency under our ADAPT framework. Extensive experiments conducted on two widely-used datasets demonstrate the superior performance of our framework in legal judgment prediction, particularly when dealing with complex and confusing charges.

## introduction legal judgment prediction (ljp) is a key research area within the legal natural language processing (nlp) community, aiming to provide automated reference judgments to help judges and other professionals manage cases more efficiently @xcite @xcite . the main challenges in enhancing judgment prediction systems are twofold: understanding case facts and distinguishing between similar charges. understanding case facts involves extracting key information from complex descriptions @xcite , while distinguishing between charges requires identifying the correct labels among confusing options @xcite . to tackle these issues, researchers have explored the use of advanced language models, the incorporation of external legal knowledge, and the reference to precedents to enhance model performance @xcite . recently, large language models (llms) have achieved state-of-the-art performance across a range of tasks due to their expanded parameter scales and training data @xcite @xcite . these models also display emergent abilities, such as instruction following and in-context learning, allowing them to quickly adapt to specific tasks based on minimal instructions or examples @xcite . although preliminary evaluations of mainstream llms in legal judgment prediction have been conducted, results indicate that their performance still lags behind many traditional supervised methods, suggesting that the existing llms are still far from being good judgment predictors @xcite @xcite . through our experiments, we find that current llms still struggle significantly more with distinguishing "confusing charges", which refer to charges that have similar or even overlapping key behaviors with other charges. this highlights a critical aspect of the ljp task: certain criminal behaviors can satisfy parts or even all of the conditions for multiple charges, creating ambiguity. due to the lack of extensive domain knowledge and reasoning training in the legal context, existing llms exhibit insufficient reasoning capability to effectively differentiate between similar charges. to adapt llms for effective legal judgment prediction, we introduce the ask-discriminate-predict (adapt) reasoning framework in this paper. our framework is inspired by the thought process of human judges, who use legal knowledge to navigate between facts and norms, as described by the classic phrase: "the gaze shuttles back and forth between facts and norms" @xcite . specifically, in the first step, ask, we decompose the noisy case description into multiple aspects under legal theory to clarify the key criminal facts. in the second step, discriminate, the model uses its parameterized knowledge to generate a candidate pool of the most probable charges and relevant law articles. within this pool, the model further differentiates among the candidates, assessing the degree of alignment between each candidate and the criminal facts. finally, in the predict step, the model synthesizes the previous reasoning process to provide the final prediction. experimental results show that our adapt prompting framework outperforms traditional prompting methods, such as direct reasoning and legal syllogism (figure 1 ), in leveraging llms for legal judgment prediction. however, llms still struggle to consistently generate accurate adapt reasoning trajectories so as to finally make correct predictions. we hypothesis that this limitation is due to their narrow legal-specific knowledge and lack of familiarity with our specialized adapt reasoning patterns. furthermore, we find that current llms frequently avoid providing sentencing ranges because of strict safety alignment protocols, which hinders their ability to complete this essential task in legal judgment prediction. to address these issues, we further propose fine-tuning an enhanced llm within our adapt framework for more comprehensive, efficient, and effective legal judgment prediction. specifically, we strengthen the llm by incorporating additional context labels-such as discriminative labels, charges, legal articles, and sentencing ranges-and prompt it to generate high-quality synthetic reasoning trajectories tailored to our adapt famework. we then use these multi-task synthetic trajectories to fine-tune a smaller llm, enabling it to perform accurate reasoning under our adapt framework. we conduct extensive experiments on two datasets, cail2018 and multiljp, which belong to single-defendant and multi-defendant scenarios, respectively. the results show that our approach achieves new state-of-the-art in both scenarios, especially on the most challenging set of charges. our contributions are summarized as: ## related work legal judgment prediction legal judgment prediction is a long-standing legal nlp task. the evolution of this task's technology has transitioned through various phases: initially relying on rule-based approaches @xcite , advancing to statistical machine learning techniques @xcite , and currently dominated by deep learning methodologies @xcite @xcite . additionally, incorporating domainspecific legal knowledge @xcite or precedents @xcite is also an important direction in existing research. with the continuous advancement of methods, this task has expanded from a simplified multi-class classification problem to complex scenarios that mirror real-world situations, such as dealing with multiple defendants @xcite and multiple law articles @xcite . in this context, we legal elements subject: "without special occupation…" criminal behaviors & consequences: "he leverage his official…" object: "which violates the victims' …" subjective: "he premeditated the crime.." case content candidate pool of charges evaluate degree of consistency distinguish confusing charges fact description: leveraging his official position, zhao utilized other's deposit certificate to withdraw money from the bank for personal consumption, lending activities… defendant name: zhao llms with adapt fraud prediction step1:ask step2: discriminate step3: predict fraud of financial voucher contract swindling … "…by fraudulently utilizing deposit certificates…" "he used forged financial documents…" "his fraud didn't involve contracts…" … "although his behavior also meets the definition of fraud, fraud of financial voucher is more applicable as a special crime…" the final prediction: crime: fraud of financial voucher; article: 194 explore the use of large language models as the foundation model and conduct robust reasoning under the setting of multi-label classification. reasoning skills in language models recent studies have shown that effective reasoning can be achieved in llms by using prompting techniques @xcite @xcite . in the legal context, previous study shows that legal syllogism can enhance the performance of ljp @xcite . furthermore, some research attempts focus on distilling the reasoning processes of large models into smaller models, thereby achieving approximate reasoning capabilities at a lower cost @xcite . our approach synthesizes the trajectory of the adapt framework to fine-tune a 7b model, enabling it to achieve both robust and efficient reasoning. ## methodology existing llms mostly rely on direct answering or judicial syllogism for reasoning, which requires the model to directly provide the correct law articles and charges. this exceeds the capabilities of these language models and even human experts, leading to bad performance. in this work, we enable llm to emulate the reasoning pattern of real-world judges to conduct discriminative reasoning for accurate legal judgment prediction. ## conclusion in this paper, we presented a novel framework to enable discriminative reasoning in llms for legal judgment prediction. our adapt framework effectively distinguishes confusing charges by determining the degree of alignment between each candidate charge and the criminal facts before the final prediction. furthermore, we utilize multi-task instruction tuning on synthetic data to enhance the language model's comprehension of this reasoning pattern. extensive experiments demonstrate the effectiveness of our approach, particularly its robustness in handling confusing charges. we believe that our work will improve the integration of llms in legal judgment prediction and contribute to the community's understanding of reasoning patterns in specific tasks.