LAR - ECHR : A New Legal Argument Reasoning Task and Dataset for Cases of the E uropean Court of Human Rights (2024)

Abstract: We present Legal Argument Reasoning (LAR), a novel task designed to evaluate the legal reasoning capabilities of Large Language Models (LLMs). The task requires selecting the correct next statement (from multiple choice options) in a chain of legal arguments from court proceedings, given the facts of the case. We constructed a dataset (LAR-ECHR) for this task using cases from the European Court of Human Rights (ECHR). We evaluated seven general-purpose LLMs on LAR-ECHR and found that (a) the ranking of the models is aligned with that of LegalBench, an established US-based legal reasoning benchmark, even though LAR-ECHR is based on EU law, (b) LAR-ECHR distinguishes top models more clearly, compared to LegalBench, (c) even the best model (GPT-4o) obtains 75.8% accuracy on LAR-ECHR, indicating significant potential for further model improvement. The process followed to construct LAR-ECHR can be replicated with cases from other legal systems.

## introduction the rise of large language models (llms) has impacted several sectors, including the legal one. in the united states, llms are being integrated into legal research and writing tools designed for both professionals and laypeople. 1 these advances are largely due to the effort of legal experts who contributed significantly in dataset development and manual evaluation @xcite . their involvement, however, is extremely costly, hence methods to construct and evaluate legal benchmarks semi-automatically are required. legalbench @xcite is an example of a widely used legal reasoning benchmark. it consists of data for 162 tasks, hand-crafted by legal experts, that evaluate six types of legal reasoning motivated by the observations above, we introduce a novel task ( §2), legal argument reasoning (lar), designed to evaluate the legal reasoning skills of llms. the task requires selecting the cor-rect next statement (from multiple choice options) in a chain of legal arguments from court proceedings, given the facts of the case. we have prepared a lar dataset for eu law, lar-echr ( §3), using court arguments from the european court of human rights (echr). it contains 403 instances; we evaluate two closed-weight and 5 openweight llms on lar-echr ( §4), comparing their performance and rankings against two legal benchmarks: legalbench and mmlu-law. for completeness, we also report scores on two general reasoning benchmarks: the entire mmlu (mmlu-full) and hellaswag @xcite . we find that: (a) the ranking of the models on lar-echr is aligned with that of legalbench, even though the two benchmarks are from different legal systems (us, eu); (b) lar-echr provides clearer distinctions among top models, compared to legalbench; (c) even the best model (gpt-4o) obtains only 75.8% accuracy on lar-echr, similar to the top accuracy on legalbench (73.3%), indicating significant potential for further model improvement. our contributions are three-fold: (i) we introduce a novel task, legal argument reasoning (lar), to evaluate the legal reasoning abilities of state-of-the-art llms; (ii) we create and publicly release lar-echr, a challenging eu-specific lar dataset; (iii) we present a method to create lar datasets for any other legal system using court proceedings with annotated arguments. ## the lar task we introduce legal argument reasoning (lar), a novel task to evaluate the legal reasoning abilities of llms. the llm is fed with the facts of the case (a list of sentences summarizing the events considered, see lar requires various types of legal and commonsense reasoning that extend beyond simple pattern recognition or memorization. as demonstrated by hellaswag @xcite , even predicting the next sentence in a generic corpus can be challenging and requires reasoning skills. in the legal domain, the complex terminology and inferences make the task of predicting a legal professional's next argument significantly harder ( §4). the arguments embody the court's legal reasoning for its decisions. as @xcite state: "[j]udges usually offer explanations which serve to justify their decisions with reference to the facts found to be established and the relevant law. [these explanations] involve the exercise of legal reasoning". merely understanding the legal terminology is insufficient, as the the facts and the relevant law must also be considered. ## the lar-echr dataset the lar-echr dataset contains arguments from the proceedings of the european court of human rights (echr). an echr court decision typically begins with the facts of the case, followed by the 'law' section, where the arguments of the parties and the court are presented, followed by the court's conclusion (e.g., verdict, fines). to create the dataset, we used statements from 'law'. to ensure that lar-echr is challenging and effectively evaluates legal reasoning, we select appropriate arguments based on criteria derived from the annotations of the legal argument mining echr (lam:echr) dataset @xcite ( §3.1). the criteria we use are described in §3.2 below. instead of generating the distractors using a language model, as in hellaswag @xcite and legallens @xcite , we draw them from similar echr cases using an algorithm we developed ( §3.3), to avoid introducing language model biases and hallucinations. lar-echr is based on 191 echr court cases. from the 191 cases, we derive 403 samples (like the example of in our experiments, we use only the test set, but we release the full dataset for future research. 2 below we describe in detail how the dataset was created. ## related work the lar task was inspired by the continuation task introduced by swag @xcite and later improved by hellaswag @xcite . it is a multiple-choice task where the model has to select the most likely continuation of an event description, such as "a woman sits at a piano" is followed by "she sets her fingers on the keys". the corpus is collected from various online sources such as wikihow 7 . similarly to lar, hellaswag is constructed automatically, via a technique called adversarial filtering (af) which selects the most persuasive llm-generated continuations as incorrect options. it is shown empirically that accurately predicting the correct continuation of an event in hellaswag requires skills that are closely related to commonsense reasoning. the primary differences with our work, aside from our focus on the legal domain, are: (a) we employ official content from court proceedings instead of events from online articles of varying credibility, (b) we use (based on the respective annotations) the most appropriate chain of arguments, and (c) we utilize human-generated challenging distractors. our dataset builds on top of two previous works: lam:echr @xcite and ecthr b @xcite . lam:echr annotated, with the help of legal experts, the arguments of 373 echr decisions with actor and argument type labels, and trained and evaluated their robertabased models on both tasks. in ecthr b the goal is to predict the articles of echr that were allegedly violated, given the facts of the case. to create lar-echr we aligned the common instances of models lar-echr (complete facts) gpt-4o 77.9 (+2.1) gpt-4o-mini 64.3 (+2.7) llama-70b 73.3 (+6.1) llama-8b 58.0 (+3.9) in the broader nlp landscape, several datasets have recently emerged for evaluating the few-shot learning capabilities and advanced reasoning skills of llms, replacing glue and superglue as the most widely used benchmarks. these new benchmarks are more aligned with the skills required by chatbot assistants designed to solve a wide range of tasks by following instructions, primarily through generating text rather than predefined labels. the most prominent of these is the massive multitask language understanding (mmlu) benchmark @xcite , which is preferred for evaluating the knowledge and general capabilities of llms @xcite . it is a multiple-choice dataset that covers 57 tasks across diverse academic subjects, three of them being law-specific. arc @xcite is another multiple-choice question-answering dataset that includes science questions requiring various types of reasoning. big bench @xcite is a challenging dataset of 204 tasks that focuses on various topics among them arithmetic, logical, common-sense and algorithmic reasoning as well as language understanding and world knowledge. inspired by the success of these benchmarks, several benchmarks for the legal domain, with the same orientation, were also made available. for example, the largest legal reasoning benchmark (in terms of number of tasks) is legalbench @xcite , comprising 162 tasks that cover six different types of legal reasoning and focus on the us legal system. @xcite created a challenging legal reasoning dataset by adapting questions from a textbook on us civil procedure which however is cast as a binary classification task and does not focus on continuations such as lar-echr. a few datasets that are focused on other legal systems than the us were also made available. for example, lawbench @xcite consists of 20 tasks on chinese law that evaluate legal knowledge understanding of llms. il-tur @xcite covers a wide range of multilingual legal text understanding and reasoning tasks for english and 9 indian languages. our dataset, lar-echr, differs from the datasets in these benchmarks in that (a) it uses the legal reasoning chain of the arguments of the judges, (b) it refers to eu law and (c) instances are collected semi-automatically from court proceedings using annotations, not handcrafted by legal experts. ## conclusion in this study, we introduced lar, a legal reasoning nlp task that requires selecting the correct next ar-gument made by judges in a case. we constructed a dataset for this task, called lar-echr, using cases from echr. we evaluated seven generalpurpose llms from three families on this dataset. the best model obtained 75.8% accuracy, indicating significant potential for further model improvement. model rankings were identical with those of legalbench, even though the datasets are based on different legal systems. despite that weak models obtained a substantially lower score in legalbench, lar-echr distinguished the top models more clearly. the process followed to construct lar-echr can be replicated with cases from any court proceedings, even from different legal systems.