The Legal Argument Reasoning Task in Civil Procedure (2022)

Abstract: We present a new NLP task and dataset from the domain of the U.S. civil procedure. Each instance of the dataset consists of a general introduction to the case, a particular question, and a possible solution argument, accompanied by a detailed analysis of why the argument applies in that case. Since the dataset is based on a book aimed at law students, we believe that it represents a truly complex task for benchmarking modern legal language models. Our baseline evaluation shows that fine-tuning a legal transformer provides some advantage over random baseline models, but our analysis reveals that the actual ability to infer legal arguments remains a challenging open research question.

## introduction arguing a legal case is an essential skill that aspiring lawyers must master. this skill requires not only knowledge of the relevant area of law, but also advanced reasoning abilities, such as using analogy arguments or finding implicit contradictions. despite recent significant contributions aimed at setting objective benchmarks for modern nlp models in various areas of legal language understanding @xcite , there is still no complex task dealing with argument reasoning in legal matters. in this paper, we propose a new task and provide a new benchmark dataset. we believe that a genuinely difficult task, coming from legal education, will help to demonstrate both the capabilities and the limitations of the current state-of-the-art legal transformation models, such as legal-bert @xcite . in particular, we present a new, publicly available 1 legal corpus for binary text classification of u.s. civil procedure problems. the goal is to classify whether a solution to a given question is correct or incorrect. the data 1 see the data sheet in the appendix for details. for the corpus is based on the the glannon guide to civil procedure by joseph glannon @xcite , which is aimed at law students. the book allows for the study of basic u.s. civil procedure topics and also contains multiple-choice questions on civil procedure problems to test the reader. with this newly created corpus, we also intend to investigate the performance of the different approaches and establish baselines and an error analysis. all source codes used to parse, extract and reformat the data and evaluate the solution methods can be found at https://github.com/trusthlt/ legal-argument-reasoning-task . legal question answering and legal reasoning in legal nlp, the number of publicly available corpora is considerably smaller, especially in the areas of argumentation and reasoning. an early compilation of legal questions in a multiplechoice format is found in @xcite . each of the 100 questions taken from the united states multistate bar examination is interpreted as an entailment task with one correct answer (entailment) and three incorrect answers (nonentailment). however, their evaluation shows that a mere similarity between theory (question) and hypothesis (answer) is insufficient to solve this task. a follow-up work extends the bar exam corpus with older exam practice questions and reformulates the task into an "answer-sentence-selection" task instead of textual entailment @xcite . additionally, 15 of the questions are annotated with an explanation. pursuing a different approach, holzenberger et al. ( 2020 ) try to answer natural language questions in the domain of tax law by encoding knowledge as a set of rules with the help of a prolog solver. holzenberger and van durme (2021) extend the previously introduced corpus and subdivide statutory reasoning into a sequence of smaller tasks. although this logic-based approach facilitates the answering, instantiation from natural language is not a trivial task. apart from resources in english, @xcite compile a corpus for legal qa in chinese with knowledge-driven questions and case-analysis questions from the national judicial examination of china. they identify different types of reasoning needed to answer the questions and conclude that existing models lack reasoning ability, especially for knowledge-driven questions. contests such as the coliee competition @xcite also include legal question answering tasks in a format that requires retrieval of relevant articles and entailment. the corpus for these tasks is taken from japanese bar exams and manually translated into english. although there are several resources available that deal with legal question answering, only few of them target statutory law in the en-glish language. there is also still a shortcoming of datasets dedicated to argumentative answers in legal qa which we aim to complement with our work. ## dataset construction we built the dataset based on the content of the glannon guide to civil procedure @xcite . we collaborated with the author and the publisher foot_0 and negotiated a permission agreement under which the resulting dataset will be freely available to the research community. the book contains 25 chapters with multiple choice questions. each chapter deals with a specific topic and asks and answers several questions. the topic of a question is introduced beforehand in an introduction. each question is followed by three to five answer candidates, where only one candidate is correct. the answer candidates can be considered as hypotheses. choosing the correct answer requires examining if the various prerequisites of a hypothesis hold. whether an answer is correct or incorrect is then discussed in the analysis afterward. we parsed the book first fully automatically, as the structure allowed us to extract the components of each instance in the resulting dataset (introduction, question, answers, analysis). anomalies in the structure, e.g., the same introduction for two questions, were caught by additional parsing rules. however, minor portions of the book had to be extracted manually, for instance the correctness of the answer candidate because the solution is addressed within the analysis' free text. the analysis is loosely designed as follows. each paragraph deals with an answer candidate and classifies it as true or false. therefore, we decided to further separate the analysis to isolate the relevant aspect for each answer. there were no keywords or structure artifacts indicating where to split the text. furthermore, several inconsistencies regarding the structure exist. thus, separating the analysis had to be done manually too. two complete examples (one incorrect and one correct, labeled as 0 and 1, respectively) are shown in appendix a. separating the analysis allows the creation of a binary classification task, which should be suitable for many application scenarios. the final legal argument reasoning task is defined as follows: task given a question with a possible correct answer and a short introduction to the topic of the question, identify if the answer candidate is correct or incorrect. after parsing the book, each question and answer pair is ordered as follows: 1. question; 2. answer; 3. solution; 4. analysis; 5. complete analysis; 6. introduction. glannon (2018) intended to ask more difficult questions at the end of each chapter. we thus created a data split accordingly. the rational data split divides the first 80% of questions of each chapter into the train set, the following 10% of questions into the dev set, and the last 10% (which tend to be more difficult) into the test set. the final dataset consists of 918 entries. to evaluate if there is any evidence that some answers were more likely to be the correct answer than others, a distance analysis using sentence-bert 3 (reimers and gurevych, 2019) was applied. this evaluation method was applied to the train-, dev-and test-set. the results indicate that there is little to no evidence that clues exist in the train and dev set. the test set shows an increased accuracy at guessing the correct answer, which is 9.88% more than the expected result of 24.5% (calculated average of guessing the correct answer of a question). ## baseline experiments a first baseline evaluation of the task included classification through pre-trained transformer models. to evaluate the performance, the macro f 1 score is used. we chose legal-bert as the classification model because of its legal tech application domain (although the model was not pre-trained with american civil procedure data). we evaluated bert @xcite as well as legal-bert (chalkidis et al., 2020) with and without fine-tuning giving it the question, answer, and introduction on an instance as input. another evaluation with these models included only the question and answer as input. for fine-tuning we experimented as suggested by @xcite with the learning-rate, weight-decay and the dropout-rate. we finished training through early-stopping (patience was set to 3). the deep learning approach uses two techniques to bypass the maximum token limit problem. sliding window simple (sws) separates the concatenated question and introduction into chunks. each chunk is then classified, and the result is the average of the predicted outputs. sliding window complex (swc) divides the introduction into multiple chunks, where each chunk contains the complete question and is padded up with the introduction. each chunk is then classified where the result is the average of the predicted outputs. ## analysis and discussion understanding legal argumentation is not an easy task by any means. therefore it is not surprising that the performance of the transformer model is struggling. the glannon guide to civil procedure is an educational book to help students learn civil procedure questions. thus, even professionals have problems answering case law questions. question: 14. additions and objections. in july 2006, a week before the three-year statute of limitations passes, carson sues herrera in federal court for breach of a contract to design a computer system for his store in calpurnia, illinois. in july 2007, he moves to amend his complaint to add a claim for violation of the state consumer protection act, based on the same dispute. the consumer protection act has a two-year statute of limitations. answer 1: the second claim would not be barred by the limitations period, as long as the judge grants the motion to amend. answer 2: the second claim would "relate back" to the date of the original filing of the case, and therefore would not be barred by the statute of limitations. answer 3: the second claim will be barred by the limitations period, because it will not "relate back" to the original filing under rule 15. answer 4: the amendment will be barred, even if it relates back to the filing of the original complaint. a comparison to a human baseline would be beneficial to evaluate the overall performance of our model. we leave establishing the human upper bound for future work. we did a brief error analysis by comparing the classification results between the fine-tuned bert and legal-bert model. out of 91 samples, the bert model labels 6 of them as correct, while the legal-bert model labels 21 as correct. the bert model predicted 3 of the 18 correct samples correctly, the legal-bert model predicted 9 of them correctly. 17 answers have divided model predictions. we read these samples for the error analysis, to understand the prediction. we assume that the legal language used in the data the legal-bert model is pretrained on, has an impact on the prediction results. this could be additionally indicated by the low amount of samples which are labeled as correct by the bert model. we further noticed that some questions have multiple answers that the legal-bert model considers correct, even though the assertion of these answers differs. one example can be seen in figure 1 . in an attempt to understand why the fine-tuned model has labeled all answers as correct we tried to follow the classification process through the usage of captum, a pytorch model interpretability library @xcite . captum is used to calculate the attribution of each word vector as input feature for the final prediction. however, as visualized in appendix c, a similar pattern for labeling each answer could not be found, despite the similarity of vocabulary between the answer possibilities. inconsistencies like these reveal that the legal-bert model does not comprehensively reason about the answer. another explanation for the shortcomings of the evaluated models could be their inherent structure. in our dataset, concatenating the question, answer and introduction leads to 689 (646, 835) words or 3508 (3243, 4245) characters on average for the rational data split. although the sliding window methods mitigate the token limits of bert, a model that can deal with longer documents, like longformer @xcite or big bird (zaheer et al., 2020) could prove to be more efficient. while a pretrained version of the longformer architecture based on legal input exists in chinese @xcite , to the best of our knowledge there is no english equivalent available. the computationally expensive pretraining of such a model and testing it with our new dataset is left for future work. even more so, we have not included the most distinguishing property of our dataset in the experiments: the analysis. it is used to explain in human language why the answer to a question is correct or incorrect. as a possible future task, it would be interesting to see if this explanation could be used to boost the reasoning capabilities of a model. ## conclusion we present a new challenging nlp task whose solution requires deeper knowledge and reasoning skills. we compare multiple transformer baselines and provide an error analysis showing that the correct prediction of the model for one instance does not prevent incorrect predictions for other relevant instances. we have obtained a license to share the dataset from the author of the original book and its publisher, and hope that it will help advance research in the complex field of legal argument reasoning. oana-maria camburu, tim rocktäschel, thomas lukasiewicz, and phil blunsom. 2018. e-snli: natural language inference with natural language explanations. in advances in neural information processing systems, volume 31. curran associates, inc. ilias chalkidis, manos fergadiotis, prodromos malakasiotis, nikolaos aletras, and ion androutsopoulos. 2020. legal-bert: the muppets straight out of law school. in findings of the association for computational linguistics: emnlp 2020, pages 2898-2904, online. association for computational linguistics. ilias chalkidis, abhik jana, dirk hartung, michael bommarito, ion androutsopoulos, daniel katz, and nikolaos aletras. 2022. lexglue: a benchmark dataset for legal language understanding in english. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 4310-4330, dublin, ireland. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. biralatei fawei, adam wyner, and jeff pan. 2016. passing a usa national bar exam: a first corpus for experimentation. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3373-3378, portorož, slovenia. european language resources association (elra). timnit gebru, jamie morgenstern, briana vecchione, jennifer wortman vaughan, hanna wallach, hal daumé iii, and kate crawford. 2021. datasheets for datasets. commun. acm, 64(12):86-92. joseph w glannon. 2018. the glannon guide to civil procedure: learning civil procedure through multiple-choice questions and analysis, 4th edition. aspen publishing. ivan habernal, henning wachsmuth, iryna gurevych, and benno stein. 2018. the argument reasoning comprehension task: identification and reconstruction of implicit warrants. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1930-1940, new orleans, louisiana. association for computational linguistics. nils holzenberger, andrew blair-stanek, and benjamin van durme. 2020. a dataset for statutory reasoning in tax law entailment and question answering. in proceedings of the natural legal language processing workshop 2020 co-located with the 26th acm sigkdd international conference on knowledge discovery & data mining (kdd 2020). nils holzenberger and benjamin van durme. 2021. factoring statutory reasoning as language understanding challenges. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 2742-2758, stroudsburg, pa, usa. association for computational linguistics. lifu huang, ronan le bras, chandra bhagavatula, and yejin choi. 2019. cosmos qa: machine reading comprehension with contextual commonsense reasoning. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2391-2401, hong kong, china. association for computational linguistics. peter jansen, niranjan balasubramanian, mihai surdeanu, and peter clark. 2016. what's in an explanation? characterizing knowledge and inference requirements for elementary science exams. in proceedings of coling 2016, the 26th international conference on computational linguistics: technical papers, pages 2956-2965, osaka, japan. the coling 2016 organizing committee. adebayo kolawole john, luigi di caro, and guido boella. 2017. solving bar exam questions with deep neural networks. in proceedings of the second workshop on automated semantic analysis of information in legal texts: co-located with the 16th a incorrect example introduction my students always get confused about the relationship between removal to federal court and personal jurisdiction. suppose that a defendant is sued in arizona and believes that she is not subject to personal jurisdiction there. naturally, she should object to personal jurisdiction. but suppose further that she isn't sure that her objection will carry the day; it's a close issue, as so many personal jurisdiction issues are under minimum contacts analysis. and suppose that her tactical judgment is that, if she must litigate in arizona, she would rather litigate in federal court in arizona. what should she do? one thing she could do is to move to dismiss in the arizona state court. but that motion is not likely to be ruled on within thirty days, and if she's going to remove, she's got to do it within thirty days. so she could do one of two things: she could move to dismiss for lack of personal jurisdiction in the state court (assuming that the state rules allow her to do that) and then remove to federal court within the thirtyday period. her motion would then be pending in federal court instead of state court, and the federal court would rule on it. or she could remove the case before a response was due in state court, and then, after removal, raise her objection to personal jurisdiction by a rule 12(b)(2) motion to dismiss or in her answer to the complaint in federal court. either way, the point is that removal does not waive the defendant's right to object to personal jurisdiction. it simply changes the court in which the objection will be litigated. it is true that, after removal, the question will be whether the federal court has personal jurisdiction. but generally the scope of personal jurisdiction in the federal court will be the same as that of the state court, because the federal rules require the federal court in most cases to conform to state limits on personal jurisdiction. fed. r. civ. p. 4(k)(1)(a). i've stumped a multitude of students on this point. consider the following two cases to clarify the point. question 7. a switch in time. yasuda, from oregon, sues boyle, from idaho, on a state law unfair competition claim, seeking @xmath0250,000 in damages. he sues in state court in oregon. ten days later (before an answer is due in state court), boyle files a notice of removal in federal court. five days after removing, boyle answers the complaint, including in her answer an objection to personal jurisdiction. boyle's objection to personal jurisdiction is answer not waived by removal. the court should dismiss if there is no personal jurisdiction over boyle in oregon, even though the case was properly removed. solution 1 analysis d is the correct answer. boyle has not waived his objection to personal jurisdiction. if the federal court lacks jurisdiction over boyle, it should dismiss the case, even though it was properly removed. complete analysis there are so many ways to go astray on this issue [...]. same as in appendix a. introduction my students always get confused about the relationship between removal to federal court and personal jurisdiction. [...] same as in appendix a. c error analyis legend: negative neutral positive true label predicted label attribution label attribution score word importance 0 tensor([1]) (1.00) 1 -2.91 14 . addition ##s and objection ##s . in july 2006 , a week before the three -year statute of limitations passe ##s , cars ##on sue ##s her ##rera in federal court for breach of a contract to design a computer system for his store in cal ##pur ##nia , illinois . in july 2007 , he move ##s to amend his complaint to add a claim for violation of the state consumer protection act , based on the same dispute . the consumer protection act has a two -year statute of limitations . | 59 the second claim would not be barred by the limitations period , as long as the judge grants the motion to amend . 0 tensor([1]) (1.00) 1 -3.61 14 . addition ##s and objection ##s . in july 2006 , a week before the three -year statute of limitations passe ##s , cars ##on sue ##s her ##rera in federal court for breach of a contract to design a computer system for his store in cal ##pur ##nia , illinois . in july 2007 , he move ##s to amend his complaint to add a claim for violation of the state consumer protection act , based on the same dispute . the consumer protection act has a two -year statute of limitations . | 60 the second claim would [unk] [unk] relate back [unk] [unk] to the date of the original filing of the case , and therefore would not be barred by the statute of limitations . 0 tensor([1]) (1.00) 1 -1.71 14 . addition ##s and objection ##s . in july 2006 , a week before the three -year statute of limitations passe ##s , cars ##on sue ##s her ##rera in federal court for breach of a contract to design a computer system for his store in cal ##pur ##nia , illinois . in july 2007 , he move ##s to amend his complaint to add a claim for violation of the state consumer protection act , based on the same dispute . the consumer protection act has a two -year statute of limitations . | 61 the second claim will be barred by the limitations period , because it will not [unk] [unk] relate back [unk] [unk] to the original filing under rule 15 . 1 tensor([1]) (1.00) 1 -3.20 14 . addition ##s and objection ##s . in july 2006 , a week before the three -year statute of limitations passe ##s , cars ##on sue ##s her ##rera in federal court for breach of a contract to design a computer system for his store in cal ##pur ##nia , illinois . in july 2007 , he move ##s to amend his complaint to add a claim for violation of the state consumer protection act , based on the same dispute . the consumer protection act has a two -year statute of limitations . | 62 the amendment will be barred , even if it relates back to the filing of the original complaint .