A rab L egal E val: A Multitask Benchmark for Assessing A rabic Legal Knowledge in Large Language Models (2024)

Abstract: The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs’ legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce \texttt{ArabLegalEval} , a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the \texttt{MMLU} and \texttt{LegalBench} datasets, \texttt{ArabLegalEval} consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset’s quality. We benchmark multilingual and Arabic-centric LLMs, such as \texttt{GPT-4} and \texttt{Jais} , respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: https://github.com/Thiqah/ArabLegalEval

## introduction the development of llms has revolutionized various fields by enhancing natural language understanding and generation capabilities. however, the applicability and performance of these models in specialized domains, such as legal contexts, specially in low-and medium-resource languages such as arabic, remain active research areas. in this paper, we report on ongoing work for evaluating the proficiency of large language models in understanding and processing arabic legal texts. given the complexity and richness of legal language, especially in arabic, it is crucial to develop benchmarks that accurately assess the models' capabilities in this domain in order to guide model development. one of the key motivations for this work and benchmark is to find out the current state of llms in the arabic legal domain. thus, we benchmark a wide range of llms from proprietary multilingual llms, such as gpt-4 (openai, 2024a), to open-source arabic-centric llms, such as jais @xcite . there are perhaps two broad categories of evaluation criteria that are useful for assessing the performance of legal llms. the first category is the ability of a model to use specific regulations, facts, and data that are relevant to a particular conversation. this can be achieved by having the llm memorize specific facts, presumably by finetuning it on particular legal corpora, or more conveniently by using a retrieval-augmented generation (rag) system to retrieve information relevant to the context at hand. the second broad category of assessment criteria is related to the model's ability to exhibit logical reasoning, understand relationships between entities and events, and apply these skills to answer questions. in this initial release of arablegaleval, we include tasks to assess the legal reasoning capabilities in arabic as well as tasks to measure the ability to recall and use legal knowledge embedded in the finetuned models. much like arabicmmlu @xcite was designed to test the general reasoning capabilities of arabic llms but includes a mix of tasks with some requiring previous knowledge, the arablegaleval benchmark seeks to develop arabic legal tasks and questions derived from original arabic legal sources, in consultation with legal professionals, to test the legal reasoning capabilities of arabic llms. the benchmark also includes some high quality translations, verified by legal experts, of tasks from english legal benchmarks (legalbench) @xcite . the benchmark includes arabic legal multiple choice questions (mcqs), question & answer (qa) pairs where the relevant saudi regulations are included in the question, as well as arabic translations of tasks from legalbench. the benchmark does not assess the ability of models to retrieve specific facts and laws relevant to a particular context from external knowledge bases. while important for the successful deployment of a legal llm, these retrieval abilities will be assessed in future tasks. the primary contributions of this paper are: the development of a novel methodology for generating a legal qa dataset that can be adapted to other domains, such as finance; and arablegaleval, an initial dataset specifically designed for assessing arabic legal knowledge in large language models. ## benckmark tasks in this section, we describe the three broad task categories in the benchmark, including 10,000+ mcqs from the native arabic moj and boe documents, a set of qa from these documents, and a quality translation of a subset of the english legalbench benchmark related to consumer contracts and privacy policies. see figure 1 . we believe that the mixture of questions from native arabic documents and translated questions gives us a somewhat diverse set of tasks and allows the benchmark to test a broader set of capabilities. in addition, this allows us to test the observation that, with increasing model scale, multilingual llms can display reasoning abilities and semantic judgment in arabic as well as in they do in english @xcite . the quantitative details and human performance baseline are presented in appendix j. ## limitations the arablegaleval benchmark currently relies heavily on saudi arabian legal documents, with some tasks translated from universal benchmarks. including documents from more arabic-speaking countries would improve geographic representation. our study did not evaluate all models, which limits generalizability; future work should include a broader range of models. limited access to legal experts affected validation depth; involving more experts would improve quality control. the dataset lacks granular categorization, such as taskspecific prior knowledge, document origin, and ai-generated content labels. adding more granular metadata and task categories would aid nuanced model training and evaluation. ## conclusions and future work we are developing an arabic benchmark to evaluate llms' legal reasoning, using saudi regulations and translated legalbench problems. future plans include adding more ksa regulatory documents, court cases, and judicial decisions to enhance this benchmark and promote advancements in arabic legal ai. c mcqs generation 3.1 mcqs evaluation criteria 3.3 prompts figure 15: question answering prompt figure 16: prompt for generating mcqs using incontext examples d translation 4.1 rouge scores for un english-arabic translation 4.3 gpt-4 translation result evaluation figure 19: gpt-4 evaluation result 4.4 sample of the translated data e qa figure 21: qa model prompt figure 22: lm judge answer similarity prompt