Developing a Pragmatic Benchmark for Assessing K orean Legal Language Understanding in Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable performance in the legal domain, with GPT-4 even passing the Uniform Bar Exam in the U.S. However their efficacy remains limited for non-standardized tasks and tasks in languages other than English. This underscores the need for careful evaluation of LLMs within each legal system before application.Here, we introduce KBL, a benchmark for assessing the Korean legal language understanding of LLMs, consisting of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the Korean bar exam (4 domains, 53 tasks, 2,510 examples). First two datasets were developed in close collaboration with lawyers to evaluate LLMs in practical scenarios in a certified manner. Furthermore, considering legal practitioners’ frequent use of extensive legal documents for research, we assess LLMs in both a closed book setting, where they rely solely on internal knowledge, and a retrieval-augmented generation (RAG) setting, using a corpus of Korean statutes and precedents. The results indicate substantial room and opportunities for improvement.

## introduction large language models (llms) show remarkable performance across various tasks. for instance, gpt-4 has passeed the uniform bar exam @xcite . however, their performance is still limited beyond the standardized bar exam questions @xcite @xcite @xcite , particularly in handling legal tasks in languages other than english @xcite . this implies it is necessary to thoroughly evaluate llms before applying them to specific legal tasks and systems. previous works have developed legal benchmarks for evaluating llms primarily in english @xcite and chinese @xcite @xcite ) mostly focusing on a closed book setting where llms answer questions based solely on the knowledge stored in their parameters. considering that legal practitioners often rely on previous legal documents such as statutes or precedents to make decision, this setting is underestimate llms' capabilities in a practical scenarios. here we introduce kbl foot_0 , a benchmark dedicated to assessing llms' capability in understanding korean legal language. kbl consists of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the korean bar exam (4 domains, 53 tasks, 2,510 examples). we evaluate llms' under two settings to better reflect their real-world usage of llms. in collaboration with legal professionals, we focus on the desgin and quality assurance of the tasks moving beyond solely standardized licensure-style questions. specifically, we assess llms in scenarios where (1) they cannot access external knowledge, and (2) they can use retrievers to search for related documents. for the second setting, we prepare a korean statute corpus consisting of 11k currently active articles and municipal ordiances and rules, and utilized a corpus of 150k korean precedents released from a previous study @xcite . the results indicates that the performance of arguably the most powerful llms, such as gpt-4 and claude-3, remains limited in handling korean legal tasks. in an open book setting, their accuracy improves by up to +8.6%, but the overall performance still varies depending on the type of corpus and the llms used. this suggests significant potential for improvement not only in the llms themselves but also in the methods of document retrieval and their integration. in summary, our contributions are as follows. our datasets, the corpus for rag, and the evaluation code will be released to the community under a cc by-nc license foot_1 . ## related work many legal ai datasets has been created contributing to nlp community @xcite @xcite @xcite @xcite @xcite @xcite @xcite . here we review only a few works that are dedicated to build benchmark for evaluating llms under zero, or few-shot setting in legal domain. @xcite developed legalbench which comprises 162 legal language understanding tasks organized according to six different types of legal reasoning based on the irac framework. however, their work is limited to tasks in english legal language understanding tasks. additionally, the benchmark does not evaluate llms in a open-book setting, and llms must rely solely on their internal knowledge. @xcite compared commercial rag-based ai systems in the us legal domain using 202 examples on generative tasks and found that even the most competent system exhibited a 17% hallucination rate through human evaluation. in contrast, our research focuses on korean legal ai tasks and emphasizes scenarios where automatic evaluations are possibles. rag systems rely on various components including databases, search algorithms, and the backbone of llms. given these complexities, it becomes impractical to manually evaluate the performance of rag systems every time a component changes. therefore, we concentrate on addressing the challenges where automatic evaluation is feasible. @xcite developed lawbench that consists of 20 chinese legal tasks. they categorized the tasks into three different levels memorization, understanding, and applying based on bloom's taxonomy. kbl differs in that it handles korean legal language understanding tasks and it evaluates llms not just in a closed-book setting but also in a open-book setting by introducing accompanying legal corpus. @xcite developed lboxopen, which includes two text classification tasks, two legal judgement prediction tasks, and one summarization task in the korean legal domain. each task contains 5k-50k examples for fine-tuning language models and evaluating performance. the datasets, except for the summarization dataset, were created by semi-automatically processing korean precedents, while human-annotated examples were used for summarization. additionally, they open-sourced 150k korean precedents. kbl differs by focusing on (1) evaluating the zero-shot performance of (large) language models across (2) more diverse tasks (7 knowledge, 4 reasoning, 53 korean bar exam), using (3) expert-annotated examples created spedifically for this study. @xcite developed kmmlu, a dataset similar to mmlu @xcite tailored for korean and cultural contexts. it includes diverse language understanding tasks along with four legal ai tasks named criminal-law, law, patent, and taxation, sourced from the public service aptitude test and various korean license exams. 3 many examples pertain to licensure exams such as the petitioner police exam, the real estate brokers exam, and the occupational safety and health exam, or cover topics in social science theory and legal terminology. these tasks are generally less challenging than those found in the bar exam. in contrast, kbl consists of questions from the korean bar exam and the newly created questions designed to assess comprehensive legal knowledge and in-depth reasoning capability relevant to the practice of law. our work also differs in that it exclusively focus on the legal domains, and we have collaborated with legal professionals to develop a pragmatic and verified benchmark. additionally, we also assess llms in scenarios where they can utilize legal documents. we have also ensured that there is no overlap between the legal task examples in kmmlu and our dataset by conducting fuzzy matching between the two datasets, with the most significant matches displayed in tables 13 14 15 16 in appendix. in addition to kmmlu, several other benchmarks have been developed specifically to evaluate language models on korean natural language processing tasks. @xcite introduced klue for korean language understanding tasks. @xcite developed kobest, targeting tasks that require advanced korean-specific knowledge. son et al. (2024b) created hae-rae benchmark, which assess korean cultural knowledge. park et al. (2024) developed ko-h5 benchmark, consisting of korean translated and reviewed datasets. our work differs in that it focuses on evaluating llms on korean legal language understanding tasks. ## datasets our benchamrk consists of 7 knowledge tasks, 4 reasoning tasks, foot_3 and multiple-choice questions from the korean bar exam. all tasks are structured as question-answering task where the model must select the correct answer for given questions, similar to mmlu @xcite . the datasets were compiled using various sources, including korean precedents, statutes, bar exams, legal qa datasets from korea legal aid corporation foot_4 , etc. the korean legal system, rooted in civil law, incorporates complex historical and cultural aspects unique to korea. notably, gpt-4 has not passed the korean bar exam, despite passing the us bar exam. this highlights that creating a korean legal benchmark involves more than translating existing benchmarks; it requires developing a new system with a unique ontology. to ensure the quality of the datasets, we developed the tasks in close corporation with legal professionals, and all the answers have been verified by 8 licensed lawyers. 6 the verification took 26 hours in total reflecting the difficulty (and the quality) of the tasks constructed. during the quality assurance process, we found freely available data, often created by individuals with semi-expertise, frequently include substantial amounts of errors (up to 21% in our study), highlighting the importance of close collaboration with experts (section a.1 covers additional general lessons learned during the dataset creation process). representative examples from individual tasks are displayed in tables 3 , 4 , 5 in appendix. ## results we evaluate four open-source llms and five commercial llms across 7 legal knowledge tasks, 4 legal reasoning tasks, and 3 bar exam tasks conducted in 2024 ( ## analysis here we provide a further analysis of the korean bar exam held in 2024. this exam was chosen because: (1) it requires both deep legal knowledge and reasoning capabilities; (2) gpt-4 exhibited surprisingly low performance despite having passed the u.s. bar exam @xcite ; (3) gpt-4 was trained using data from before the 2024 bar exam, ruling out the possibility of data contamination. we first categorizes individual questions into two types: (1) rule or (2) application. a question is categorized as application-type if it requires not just an understanding of legal knowledge or principles, but also how these are applied to specific real-world cases. otherwise, it is considered ruletype. although the irac method-issue, rule, application, and conclusion (irac)-typically has four categories, we focus on two because questions often belong to multiple categories, making them difficult to clearly categorize. in the civil domain, there are 29 rule-type questions and 41 application-type questions. the criminal domain includes 4 rule-type and 36 application-type questions, while the public domain comprises 23 ruletype and 17 application-type questions. rag primarily enhances performance on ruletype questions fig. 1 depicts gpt-4's accuracy on the 2024 bar exam, highlighting performance without rag (blue), with the statute corpus (orange), with the precedent-corpus, and with both corpora (red). the left panels display accuracy on rule-type questions, while the right panels show on application-type questions. in both civil and criminal domains, using the precedent corpus significantly enhances accuracy for rule-type questions (upper left and middle left panels, blue vs green). however, there are no improvements in civil application-type questions and only marginal improvements in applicationtype questions. conversely, in the public domain, neither the precedent corpus nor the statute corpus proves beneficial. ## conclusion we propose kbl, a pragmatic benchmark designed for korean legal language understanding that comprises (1) 7 knowledge tasks, (2) 4 reasoning tasks, and (3) korean bar exams. the first two tasks were designed with close corporation with legal professionals and the answers were validated by lawyers. the results indicate that there is still a significant room for improvement in llms' capabilities in the korean legal domain. additionally, recognizing that legal research often involves consulting related legal documents, we equip llms for evaluation in a rag setting by providing two accompanying legal corpora for retrieval: a precedent corpus from a previous study @xcite and a statute corpus developed in this study. enabling llms to utilize external legal documents via a simple bm25 retriever has shown to improve performance but not always depending on several factors. this highlights the importance of our work that provides a common playground for automatic evaluating of rag systems.