A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction (2023)

Abstract: Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. However, recent disputes over GPT-4’s law evaluation raise questions concerning their performance in real-world legal tasks. To systematically investigate their competency in the law, we design practical baseline solutions based on LLMs and test on the task of legal judgment prediction. In our solutions, LLMs can work alone to answer open questions or coordinate with an information retrieval (IR) system to learn from similar cases or solve simplified multi-choice questions. We show that similar cases and multi-choice options, namely label candidates, included in prompts can help LLMs recall domain knowledge that is critical for expertise legal reasoning. We additionally present an intriguing paradox wherein an IR system surpasses the performance of LLM+IR due to limited gains acquired by weaker LLMs from powerful IR systems. In such case, the role of LLMs becomes redundant. Our evaluation pipeline can be easily extended into other tasks to facilitate evaluations in other domains. Code is available at https://github.com/srhthu/LM-CompEval-Legal

## introduction large language models have achieved great success in various natural language processing (nlp) tasks @xcite , while there are still some disputes over the potential for domain-specific applications @xcite . focusing on the law domain, the leading llm, @xcite , was claimed to pass the uniform bar exam (ube) with a 90th percentile score. although inspiring, however, this result was pointed out to be overestimated @xcite . for similar cases, "t" refers to true similar cases with the same charges as the query cases, while "f" refers to false similar cases. for task settings, "zs" is the abbreviation for zero-shot and "fs" for few-shot. this raises an interesting question: how exactly llms perform in various real-world legal tasks? in this paper, we design practical baseline solutions based on llms and systematically investigate their competency in the law, to shed light on other domains as well. we attribute the main issues of the previous benchmark as follows. first, ube is too general and not subject to any legal jurisdiction @xcite . second, ube contains multichoice questions and open-ended questions that require human experts to evaluate. to avoid human evaluation, some datasets @xcite replace open-ended questions with multi-choice questions. however, in real-world applications, there are not only multi-choice but also open questions. using multi-choice questions only may not be comprehensive enough. third, specifically in but not limited to common law @xcite , similar cases are always introduced as evidence to support expertise legal reasoning @xcite , which are not fully studied in previous benchmark @xcite . for the first issue, we choose legal judgment prediction (ljp) @xcite @xcite as the example task for investigation. it is a real-world problem to determine the charges committed by the defendants under a juridical system, as shown in figure 1 . ljp is typically formulated as a classification task to predict the most possible one from a list of predefined charges. then, for the second and third issues, we design four settings derived from two work scenarios of llms to cover open and multichoice questions and the usage of similar cases. in the first scenario, llms work alone without explicit knowledge in prompts, assuming all domain knowledge is implicitly stored in parameters. in the second scenario, llms coordinate with an information retrieval (ir) system that enriches prompts with similar demonstrations and label candidates to benefit expertise reasoning. specifically, demonstrations consist of pairs of similar cases and their charges, which are retrieved by the ir system based on similarity of case facts. labels of the retrieved cases can form label candidates, shown as circles of different colors in figure 1 , to hint llm with label information and narrow down label space @xcite . the four evaluation settings in figure 1 can be categorized based on the presence of two elements in prompts: demonstrations (similar cases) and label candidates. demonstrations convert the setting from zero-shot to few-shot prompting, while label candidates simplify the task from open questions to multi-choice questions 1 . the first scenario corresponds to the first setting, where neither element is present, while the second scenario encompasses the remaining three settings. we evaluate five upto-date llms of the close-source gpt-3 @xcite family, chatgpt and gpt-4 (openai, 2023), and open-source llms including vicuna @xcite , chatglm @xcite and bloomz @xcite . the evaluation is conducted on a chinese ljp dataset, namely cail @xcite , which contains cases of 112 criminal law charges 2 . we highlight our key findings as follows: ## baseline method the goal of legal judgment prediction is to determine the committed charges given case facts. to harness llms for ljp, we adopt in-context learning @xcite and use llms to generate the charges conditioned on prompts (section 2.1). to enhance llms, we incorporate label candidates and demonstrations consisting of similar cases into prompts, which are acquired by an ir system (section 2.2). this derives four settings of baseline solutions, namely zero-shot open questions, few-shot open questions, zero-shot multi-choice questions, and few-shot multi-choice questions. the multichoice settings employ label candidates while fewshot settings include demonstrations, as shown in figure 1 . finally, we introduce how to simulate ir systems with different capabilities to understand their effects (section 2.3). ## llm vs. llm with ir system we initially present the overall results, highlighting the importance of label candidates and similar cases, and conduct a comparative analysis of the models. subsequently, we investigate the relationship between label candidates and self-consistency to unveil their actual effects on expertise reasoning. additionally, we perform an ablation study by replacing similar cases with fixed cases as demonstrations to further understand their impact. ## paradox of information retrieval system the significance of similar demonstrations illustrated in section 4.3 has motivated research focusing on prompting-oriented ir systems @xcite to retrieve high quality demonstrations. however, we raise an intuitive question: do llms gain substantial improvement from ir systems compared to the knn baseline that harnesses ir systems for classification tasks? the question is inspired by our observation that the bm25 retriever achieves 48.03% of precision@1 foot_7 and 57.68% prediction accuracy by majority vote of top @xmath0. this observation suggests a paradoxical scenario wherein an ir system outperforms the combination of llm and ir, with the llm taking on the leading role and the ir serving as a supporting role. in such a scenario, the llm becomes redundant due to its failure to fully utilize the informative retrieved documents. to investigate the paradox, instead of experimenting with different ir systems, we manipulate the bm25 retriever to simulate a series of ir systems with different capabilities measured by precision@1 as described by section 2.3. we take a case study of chatgpt, whose 1-shot performance under different ir systems (denoted as precision@1) is shown in figure 6 . results although the performance of chatgpt enhanced by ir systems improves with ir capability, it will eventually underperform the ir system once the ir capability surpasses a certain threshold. in the ideal situation where true similar cases are always retrieved, chatgpt is unable to attain 100% accuracy and lags significantly behind the optimal ir system. according to appendix a.4, all smaller llms are not comparable to the bm25 retriever. ## discussion we compare the llms with supervised baselines. we fine-tune bert @xcite on the same training set and achieve a comparable accuracy of 68% to chatgpt but lower than gpt-4. since llms are not fine-tuned on the specific ljp task, this result highlights the remarkable superiority of llms in acquiring significant knowledge and leveraging transfer learning @xcite . however, we observe that bert's performance improves to 89% when trained with the original training set (~10k). we find that certain knowledge is present in shadow features, which can be easily learned with supervision. these superficial features can result in biased supervised models. fortunately, unsupervised pre-training objectives, make llms more robust and less vulnerable to this issue. this depicts a promising future for nlp applications in various domains. ## conclusion to address the deficiency in evaluating the competency of llms in the field of law, we focused on the task of legal judgment prediction and devised four settings to facilitate a thorough evaluation that encompassed both open and multiple-choice questions and incorporated similar cases to aid in the decision-making process. the evaluation results revealed different behaviors among the prominent llms, namely gpt-4 and chatgpt, compared to their smaller counterparts. both gpt-4 and chatgpt exhibited remarkable proficiency in effectively leveraging domain knowledge in various formats. among the smaller llms, chatglm displayed greater robustness, while bloomz showcased superior zeroshot ability. we presented an intriguing paradox wherein llms could become abundant in the presence of a powerful ir system. when improving ir systems to benefit llms, it is crucial for researchers to acknowledge this paradoxical scenario and prevent great disparity between llms and ir systems.