S em E val-2024 Task 5: Argument Reasoning in Civil Procedure (2024)

Abstract: This paper describes the results of SemEval-2024 Task 5: Argument Reasoning in Civil Procedure, consisting of a single task on judging and reasoning about the answers to questions in U.S. civil procedure. The dataset for this task contains question, answer and explanation pairs taken from The Glannon Guide To Civil Procedure (Glannon, 2018). The task was to classify in a binary manner if the answer is a correct choice for the question or not. Twenty participants submitted their solutions, with the best results achieving a remarkable 82.31% F1-score. We summarize and analyze the results from all participating systems and provide an overview over the systems of 14 participants.

## introduction we frame our task in a simple manner: classifying whether the given answer is a correct solution to the question or not. with this task, we want to put the legal reasoning capabilities of various stateof-the-art models to the test and provide a reliable benchmark. ## related work as the task is based upon our previous paper @xcite , we refer to the detailed related work section in there. in a nutshell, legal question answering is an inherently difficult task because it requires both reasoning skills and expertise. legal question datasets in nlp are scarce and vary in terms of the specific topics covered, such as the u.s. multistate bar examination @xcite , tax law @xcite , and japanese bar exams @xcite . although existing datasets focus on finding the correct answer to the question posed, the reasoning behind a correct or incorrect answer is often ignored. more recently, llms have found their way into legal question answering, demonstrating their potential in this area @xcite by solving complex legal questions at a level comparable to humans. but these circumstances also highlight the need for appropriate tasks to evaluate such systems @xcite . ## dataset the dataset was collected by parsing the glannon guide to civil procedure (glannon, 2018) which was done in our previous work @xcite . the details of the data collection and baseline methods are also outlined there. instead of treating the questions from the book as multiple choice queries, we decided to pair each answer with its question and attach a binary label for a correct or incorrect conclusion. because there are usually multiple incorrect answers to a question, the dataset is highly imbalanced towards incorrect answers. a question can either be a stand-alone sentence or in cloze text form. to make the context question 7. a switch in time. yasuda, from oregon, sues boyle, from idaho, on a state law unfair competition claim, seeking $250,000 in damages. he sues in state court in oregon. ten days later (before an answer is due in state court), boyle files a notice of removal in federal court. five days after removing, boyle answers the complaint, including in her answer an objection to personal jurisdiction. boyle's objection to personal jurisdiction is answer not waived by removal. the court should dismiss if there is no personal jurisdiction over boyle in oregon, even though the case was properly removed. solution 1 analysis d is the correct answer. boyle has not waived his objection to personal jurisdiction. if the federal court lacks jurisdiction over boyle, it should dismiss the case, even though it was properly removed. complete analysis there are so many ways to go astray on this issue [...]. introduction my students always get confused about the relationship between removal to federal court and personal jurisdiction [...]. of most of the questions clear, there is an introduction text which provides background information to the question. in addition, glannon has written further explanatory texts which justify why the answer was a correct choice or not. each data point in the dataset consists of question, answer candidate, solution, analysis (answer), complete analysis (all answers to the question), introduction. an example data point is presented in figure 1. however, the dataset version used in the competition differs slightly from the original version. to correct errors in the initial version of the dataset, we removed a mistakenly included chapter of the book. additionally, we corrected two instances in which the explanation text was missing. although the dataset size changed, the partitions are still based on the paradigm used in @xcite , resulting in a training partition (666), development partition (84), and testing partition (98). the rational data split is meant to sort questions which appear later in each chapter into the test set, assuming that these questions are harder to answer than earlier ones. to conceal the labels in the test set, we eliminated both fields label and analysis in that partition. ## task description reasoning is still one of the hardest task state-ofthe-art models and techniques can face. simply understanding language is certainly not enough to understand expert legal questions, much less answer them correctly. the task is meant to probe the capacity of methods for understanding complex legal topics and applying them in exemplary scenarios. however, to avoid over-complicating the output and evaluation, the task is formulated as a simple yes or no question. by default this approach also makes the task harder, because there is no option to find one correct answer by process of elimination. the task remains the same as introduced by @xcite : task given a question with a possible correct answer and a short introduction to the topic of the question, identify if the answer candidate is correct or incorrect. although systems may use the analysis that is provided in the training and development partitions for enhancement, they should be able to produce a prediction based on introduction, question and answer candidate alone. ## participant systems during the competition period, we received 59 requests for the dataset. of the 55 participants who registered on the codalab platform, 20 submitted results in the evaluation phase. we summarize and evaluate the 14 teams that submitted system papers. in addition to the descriptions, we present a brief summary of the key features of the proposed systems in ## analysis 6.1 error analysis we take a closer look how individual instances in the test set were classified. for this, we cluster the instances by the chapter they appear in and sort the chapters by the average performance (see figure 2 ). with the goal of identifying the questions that were more challenging for the systems to answer, we cross-check the chapter titles and content of the best and worst-performing chapters. chapters 6, 12, and 7 were the best-performing and cover the topics "more personal jurisdiction: general in personam jurisdiction and in rem jurisdiction", "two ways to run a railroad: substance and procedure after york, byrd, and hanna" and "more than an afterthought: long-arm statutes as a limit on personal jurisdiction". legal expertise would be required to carefully assess why some chapters appear more difficult than others. throughout our analysis, we could not identify a clear common factor for difficult and easy instances. this can be attributed to the small sample size of the test partition and the carefully designed questions. please refer to ## conclusion in this paper we presented an overview of task 5 of the semeval-2024 competition, a task on argument reasoning in civil procedure. the dataset and the problems related to data leakage due to partitioning were briefly outlined. the submitted systems were described and summarized, and insights into the achieved results were provided. the submitted solutions indicate that llms, specifically gpt-4, are surprisingly decent in handling argument reasoning in civil procedure. although legal-bert and other older domain-specific models can still solve the task to some extent, they are outperformed by a significant margin. the average performance of older or simpler techniques also suggests that this task is a suitable benchmark for evaluating legal reasoning in civil procedure. although the top-performing systems still have room for improvement, the submitted solutions demonstrate that performance can be enhanced using various techniques. this task is far from solved. a future iteration of this competition could also utilize the mostly unused analysis field. this could alleviate the dataset's shortcoming of lacking traceable reasoning steps in the solution to further boost the emphasis on the reasoning aspect of the task.