L aw B ench: Benchmarking Legal Knowledge of Large Language Models (2024)

Abstract: We present LawBench, the first evaluation benchmark composed of 20 tasks aimed to assess the ability of Large Language Models (LLMs) to perform Chinese legal-related tasks. LawBench is meticulously crafted to enable precise assessment of LLMs’ legal capabilities from three cognitive levels that correspond to the widely accepted Bloom’s cognitive taxonomy. Using LawBench, we present a comprehensive evaluation of 21 popular LLMs and the first comparative analysis of the empirical results in order to reveal their relative strengths and weaknesses. All data, model predictions and evaluation code are accessible from https://github.com/open-compass/LawBench.

## introduction legal tasks encompass a broad spectrum of applications, predominantly text-based, necessitating comprehension and interpretation of highly professional legal text. currently, they are primarily conducted by legal experts, who require years of extensive specialized training to process legal cases. endowing large language models (llms) with legal expertise can not only improve the working efficiency of legal officers, but also address the overwhelming demand of legal assistance from non-professionals, thereby improving public access to justice @xcite . there have been works assembling legal-related tasks, including lexglue @xcite , lbox open @xcite , lextreme @xcite , scale @xcite , and legalbench @xcite . legalbench, in particular, presented the first steps towards constructing an interdisciplinary collaborative legal reasoning benchmark for the english language and evaluated 20 llms in 162 legal tasks. while these benchmarks have been developed for legal systems * equal contribution, † corresponding author. unlike the common law system, which is widely accepted in the western countries, the chinese legal system is rooted in the civil law family. judges in the civil law system are obliged to respect the established statutory law articles and ground their decisions on them. understanding and applying existing statutes and codes, rather than studies of precedents, are of paramount importance @xcite . hence, models that perform well in the aforementioned benchmarks may not necessarily perform well on the chinese legal tasks, and it is necessary to develop a benchmark that emphasizes the required skill set for the chinese legal system. with this in mind, we present lawbench, the first evaluation benchmark composed of 20 tasks aimed to assess llms' capabilities on performing legalrelated tasks under the chinese civil law system. alongside lawbench, our contributions in this paper are three-fold. first, we develop a taxonomy that divides these 20 tasks into three skill levels according to widely accepted bloom's cognitive taxonomy @xcite , providing suggestive evidence that lawbench is capable of assessing an llm's ability to memorize, understand, and apply legal knowledge. second, we perform an extensive evaluation of 21 popular llms on lawbench to assess their ability to perform chinese legal tasks. third, we conduct the first comparative analysis of these results in order to gain insights into the relative strengths and weaknesses of the llms. finally, to stimulate work in this area of research, we integrate the benchmark and evaluation code into the opencompass platform @xcite . ## related work in this section, we focus our discussion on existing benchmarks developed for the legal domain. chinese benchmarks. our benchmark, law-bench, is the first evaluation benchmark developed for the chinese legal domain. lawbench has influenced the design of a subsequently released benchmark, laiw @xcite , according to its authors. among the 14 tasks in laiw, six can be found in lawbench, three are similar to a task in lawbench, and five are not present in lawbench. 1 a few points deserve mention. first, unlike law-bench, laiw does not contain any legal knowledge memorization tasks (see section 3.1). second, there are a number of critical legal application tasks that are present in lawbench but not laiw, including tasks that involve proofreading a legal document as well as tasks that involve predicting an amount (e.g., the monetary penalty) associated with a case. we believe that the failure to include such application tasks will disallow laiw to evaluate a model in certain important scenarios. disc-law-eval @xcite is another chinese legal benchmark that is primarily focused on assessing a model's capabilities on legal exams, with an emphasis on multiple-choice questions. in addition evaluating a model's performance on multiple-choice questions, lawbench examines a model's proficiency in other tasks such as information extraction and reading comprehension, and ensures that the scope of testing is not confined to skills required for judicial examinations. non-chinese benchmarks. other prominent legal benchmarks include lexglue @xcite , which focuses on eu and american laws, lbox open @xcite , which focuses on south korean laws, legalbench @xcite , which focuses on american laws, lextreme @xcite , which covers 24 indo-european and uralic languages, and scale @xcite , which covers three swiss languages across a range of legal tasks. these benchmarks are designed to evaluate the legal logic of a model when used with common law jurisdictions, which differs significantly from civil law jurisdictions, such as the one adopted in china. 2 as a result, they have distinct application scenarios and processes. in chinese judicial practice, tasks such as document review and dispute focus detection are crucial but are not present in these benchmarks. given the uniqueness of the chinese legal system and the diversity of the tasks in lawbench, we believe lawbench will contribute to the multilinguality of global legal research. we conclude this section by noting that in the aforementioned papers, model evaluations on the benchmarks are often accompanied by shallow or even no analysis of the results. for instance, we consider the analysis in the legalbench paper shallow because it was solely performed based on the average performance of the models over all tasks. in contrast, a key contribution of our work is a comparative analysis of the models not only w.r.t. their average performance but also at the task level. ## lawbench construction in this section, we provide a detailed description of the principles behind the design of lawbench. ## conclusion we presented lawbench, the first benchmark composed of 20 tasks that is aimed to assess llms in performing legal-related tasks under the chinese civil law system. we provided a structured taxonomy of the skill set required for legal-related tasks through which the 20 tasks were divided into three groups that correspond to three cognitive dimensions in bloom's taxonomy, namely memorization, understanding and application. we assessed the performance of 21 llms on the 20 tasks. our results demonstrated that current llms were still unable to give meaningful judicial aid, and their scores on most tasks were often poor. nevertheless, we conducted the first comparative analysis of the results in an attempt to reveal the strengths and weaknesses of these llms on the 20 legal tasks. our analysis revealed that while fine-tuning opensource llms on legal data resulted in some improvements, they still offered inferior performance to gpt-4. as the legal field is highly professional, much of the data used in practical applications is confidential. therefore, we believe that the release of lawbench and our analysis can serve as a solid foundation for future research in this area.