Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement (2023)

Abstract: Generative language models (LMs) are increasingly used for document class-prediction tasks and promise enormous improvements in cost and efficiency. Existing research often examines simple classification tasks, but the capability of LMs to classify on complex or specialized tasks is less well understood. We consider a highly complex task that is challenging even for humans: the classification of legal reasoning according to jurisprudential philosophy. Using a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts, we systematically test the performance of a variety of LMs. We find that generative models perform poorly when given instructions (i.e. prompts) equal to the instructions presented to human annotators through our codebook. Our strongest results derive from fine-tuning models on the annotated dataset; the best performing model is an in-domain model, LEGAL-BERT. We apply predictions from this fine-tuned model to study historical trends in jurisprudence, an exercise that both aligns with prominent qualitative historical accounts and points to areas of possible refinement in those accounts. Our findings generally sound a note of caution in the use of generative LMs on complex tasks without fine-tuning and point to the continued relevance of human annotation-intensive classification methods.

## introduction academia and industry increasingly use generative language models (lms) for document annotation and class-prediction tasks, which promise enormous improvements in cost and efficiency. however, research tends to focus on relatively simple and generic annotation contexts, such as topic or query-keyword relevance @xcite @xcite . but many potential applications call for annotation or prediction of complex or specialized concepts, such as whether a writer reflects a particular school of thought. these questions may be difficult even to describe to a trained human annotator, much less apply. it is unclear if generative lms perform well on this type of complex and specialized task. in this study we systematically examine the ability of large lms to parse a construct that is difficult even for highly trained annotators: modes of legal reasoning. we consider two prominent modes of legal reasoning that judges employ as identified by legal historians, in addition to a null or noninterpretative class. although the classes of legal reasoning identified by historians reflect relatively well-defined concepts, determining whether a particular document reflects a mode of reasoning can be exceptionally challenging. we suspect this is common to many high-value but specialized tasks, such as classifying complex emotional states or detecting indirect racial or gender bias. these tasks often require both abstract reasoning and specialized knowledge. legal reasoning is a suitable setting for examining model performance on a highly complex classification task. the foundation of our research is a new dataset of thousands of paragraphs of historical supreme court opinions annotated by a team of upper-year students at a highly selective law school. we find that even the largest models perform poorly at the task without fine-tuning, even when using similar instructions as those given to human annotators. this finding suggests that lms, even as augmented through few-shot or chain-ofthought prompting, may not be well-suited to complex or specialized classification tasks without taskspecific fine-tuning. for such tasks, substantial annotation by domain experts remains a critical component. to demonstrate this point, we examine the performance of established to cutting-edge lms when fine-tuned on our annotated data. our results show strong performance for many of these fine-tuned models. our analysis explores various approaches to model structure, such as a multi-class task versus serialized binary tasks, but we find that using an in-domain pre-trained model, legal-bert @xcite , results in the highest performance for a task that requires specialized domain knowledge. the primary contributions of this paper are as follows: in sum, our paper shows that in a complex and specialized domain, without fine-tuning, current generative models exhibit serious limitations; there is a continued need for domain-expert annotation, which can be effectively leveraged to unseen instances through fine-tuned models. foot_0 ## related work researchers have developed strategies to guide lms to perform complex tasks without the time and infrastructure costs of fine-tuning, often by breaking decisions down into multiple steps of reasoning. @xcite use few-shot chain-ofthought (cot) prompting to provide a model with examples of intermediary logic before making a decision. an alternative, zero-shot cot also results in improved performance in certain tasks, as lms are prompted to break down their reasoning (e.g. "let's think step by step") @xcite . another procedure, plan-and-solve (ps) prompting, asks a model to devise and execute a plan for reasoning through a problem @xcite . at certain tasks and with these prompting strategies, lms perform annotation or classification tasks at the level of humans. given the high costs (e.g., time, money, logistics) of collecting high-quality human-annotated data, recent work has suggested that annotation tasks previously performed by students, domain experts, or crowdsourced workers could be replicated with equal performance by lms. generative models perform well on query-keyword relevance tasks @xcite ; on topic detection in tweets @xcite ; or on detecting political affiliation in tweets @xcite . @xcite suggests that zero-shot and few-shot models are a legitimate alternative for stance detection because of the unreliability of human annotators due to the vast contextual information annotators may or may not draw from. in the legal domain, scholars examine classification performance of generative lms on the type of case (e.g., contracts, immigration, etc) (livermore and rockmore, 2023), or on the court's use of a specific canon in statutory construction @xcite . the range of applications for which generative lms might adequately perform is an open question. we have found limited work that requires specialized knowledge in addition to the use of abstract reasoning skills. in this study, we ask the models to engage in precisely this form of reasoning, which is challenging even for domain-expert humans. what distinguishes this form of reasoning is that it requires the analyst to conceptualize abstract principles and determine whether a specialized, domain-specific example fits one of those concepts. this difficulty contrasts with simpler tasks, which may key off well-established associations in training data between concepts, such as political affiliation and word usage. ## legal reasoning our focus is on legal reasoning involving statutory interpretation. 2 in the united states, congress writes statutes, but determining how statutes apply in individual cases is often left to the courts. every year, the supreme court decides numerous cases of statutory interpretation, ranging from questions such as whether a tomato is a "vegetable" or a "fruit" for the purposes of import tariffs as in nix v. hedden, 3 to whether the clean air act authorizes the environmental protection agency to regulate greenhouse gases as in west virginia v. epa. 4 jurists adopt a wide range of approaches to interpreting statutes and engaging in legal reasoning more generally. a classic distinction is between what 20th century legal scholar karl llewellyn referred to as "formal" and "grand" styles of reasoning @xcite . grand reasoning refers to a form of legal reasoning that respects precedent but is characterized by "the on-going production and improvement of rules which make sense on their face" @xcite . on interpretive questions, it therefore privileges work-ability, future orientation, and common-sense understandability. by contrast, formalism focuses not on the "policy" considerations of a law's consequences, but instead on its more mechanical application: "the rules of law are to decide the cases; policy is for the legislature, not for the courts... opinions run in deductive form with an air or expression of single-line inevitability" @xcite ). llewellyn's modes of legal reasoning apply more broadly than statutory interpretation. with respect to statutory interpretation specifically, under the grand style of reasoning "case-law statutes were construed 'freely' to implement their purpose, the court commonly accepting the legislature's choice of policy and setting to work to implement it" @xcite ; showing his sympathies, under the formal style, llewellyn wrote, "statutes tended to be limited or even eviscerated by wooden and literal reading, in a sort of long-drawn battle between a balky, stiff-necked, wrong-headed court and a legislature which had only words with which to drive that court" @xcite . though their terminology does not always follow llewellyn, other legal scholars identify a similar primary distinction in legal reasoning. horwitz, for instance, centers discussion on legal "orthodoxy," which seeks to separate law from consequences and elevate "logical inexorability" @xcite . against orthodoxy, horwitz identified a progressive critique, which "represented a broad attack on claims of classical legal thought to be natural, neutral, and apolitical" @xcite . other prominent accounts follow llewellyn's distinctions more explicitly @xcite . operative doctrines in important areas of law, moreover, reflect the broad schools of thought: e.g., the "rule of reason" in anti-trust, which involves holistically examining the pros and cons of conduct rather than a rule-like test under the sherman act, may be understood to reflect the socially-aware grand school class definition formal a legal decision made according to a rule, often viewing the law as a closed and mechanical system. it screens the decision-maker off from the political, social, and economic choices involved in the decision. grand a legal decision that views the law as an open-ended and on-going enterprise for the production and improvement of decisions that make sense on their face and in light of political, social, and economic factors. none a passage or mode of reasoning that does not reflect either the grand or formal approaches. note that this coding would include areas of substantive law outside of statutory interpretation, including procedural matters. our contribution focuses on this broad consensus around a key distinction in the modes of legal reasoning. on the one hand, a mode of reasoning that is innovative, open-ended, and oriented to social, political, and economic consequences of law; on the other hand, a mechanical, logic-oriented approach that conceives of the law as a closed and deductive system of reasoning. though scholars differ on terminology, we follow llewellyn and refer to these schools as grand and formal ( not only does this basic conceptual consensus exist, but there is also rough consensus on periodization: that is, the periods of history in which each school was dominant. the "conventional" @xcite view is that in the pre-civil war period, the grand style dominated; in the period between the civil war and world war i, the formal style dominated; the grand school then dominated for much of the twentieth century @xcite . the standard view is that we currently live in a period of formalism @xcite . we use this periodization to validate our measure; but also use the measure to provide a nuanced account of historical trends in legal reasoning. ## data we use a dataset of 15,860 historical united states supreme court opinions likely involving statutory interpretation and issued between 1870 and 2014. 5 the raw data come from harvard's caselaw access project. 6 opinion text underwent minimal pre-processing, but all case citations were removed to reduce cognitive workload for the annotators. 7 to create the dataset for annotation, we included only opinions that conduct statutory interpretation and then upsampled paragraphs likely to use formal or grand reasoning. the seed terms and details about this sampling procedure are included in appendix a. in the final collection, 25% of paragraphs include at least one formal seed, 25% include at least one grand seed, and the remaining 50% are randomly sampled. ## human annotations for legal reasoning a team of domain experts, four upper-year law students at a highly selective law school, annotated selections from court opinions as formal, grand, or lacking statutory interpretation. this team collaboratively developed and tested a codebook (included in appendix d) by iteratively annotating court opinions and calculating inter-rater reliability on a weekly basis over the spring 2023 semester. the annotation task asked each annotator to assign one of three labels, "formal," "grand," or "none," to each paragraph. a fourth label, "low confidence," could be added in addition to one of the three core labels if the type of reasoning was ambiguous. we calculated inter-rater reliability using krippendorff's alpha to evaluate agreement between the four labelers and across the three main classes. this coefficient was calculated weekly and guided the decision of when to start collecting data for training. paragraphs with high disagreement were discussed in depth and these discussions led to the revision of our codebook. we note that while this annotation is formally a three-way classification task, the low dimensional-ity of the output space does not imply that the task is easy. in fact, it took weeks for highly trained upper-year law students to reach a level of expertise at which they were able to reach consistent results. inter-rater reliability increased after the introduction of a decision chart (figure 1 ), which broke down decisions about each of the classes into a series of guided questions (appendix b). for each paragraph assigned the low confidence label, the team deliberated over possible labels until reaching a group decision through a majority vote. training and evaluation data includes this resulting label for low confidence paragraphs; not the initial label. in total, excluding paragraphs prior to decent inter-rater reliability, 2748 paragraphs were labeled and included in the training and evaluation data. even with the upsampling of legal interpretation based on seed terms, paragraphs that did not engage in legal interpretation or interpreted something other than a statute, our "none" class, made up 68% of the data ( ## automated annotation with lms though each member of the annotation team was an upper-year law student who had completed highly relevant coursework, the task remained difficult for the human annotators, as reflected in the mid-range inter-rater reliability (0.63 krippendorff's alpha). of legal reasoning were clear, but determining whether specific instances reflected one mode or another required specialized knowledge and an ability to map those abstract concepts to the incomplete evidence in the paragraphs. the complexity of this task makes it challenging for a generative model prompted in-context or with cot reasoning. as an initial experiment, we begin with a slightly simplified task: identifying whether a passage involves some form of legal reasoning (regardless of class). we then compare a larger variety of models on the primary task of interest: identifying instances of formal and grand legal reasoning. for both tasks, we compare the performance of in-context and fine-tuned models, with the expectation that identifying legal reasoning is more achievable for in-context models than identifying the specific formal or grand classes. here, we test thresholds of task complexity, to better identify the point at which an annotated dataset for fine-tuning is needed; not just a carefully crafted prompt. ## model performance we test the performance of all models on the same five test splits of data and find that the fine-tuned models consistently outperform the in-context mod- ## application: periods of legal reasoning the conventional wisdom among legal observers is that we currently live in a period in which the formal style of reasoning predominates @xcite . yet it has not always been this way: in other historical periods, the grand style of reasoning prevailed. indeed, there is a rough consensus in the legal literature regarding historical periodization @xcite . writing in the mid-twentieth century, llewellyn identified three periods of legal reasoning. prior to the civil war, the grand style of reasoning predominated; from the civil war to world war i, the formal style of reasoning prevailed; and from world war i onward, courts again operated under the grand style of reasoning. more recently, scholars identify the 1980s as a critical point of transition towards formalism @xcite . other scholars identify fundamentally similar periodizations @xcite , and though differences exist, it is possible to speak of a "conventional" view @xcite . these historical characterizations arise from leading scholars reading judicial opinions and forming judgments through the use of their full faculties about the prevailing style of reasoning. our data starts at reconstruction (the period following the us civil war) and allows us to examine the convergence between the scholarly consensus historical periodization and the historical periodization implied by our lm-derived results. we can also use our predictions to offer more granular assessments of the periods and potentially to adjudicate differences among the views of prominent scholars. this latter analysis is preliminary, in part, because earlier scholars examined judicial reasoning broadly, whereas our current analysis considers only supreme court opinions involving statutory interpretation. 10 for this exercise, we study historical trends in the predictions from the highest performing model, multi-class, fine-tuned legal-bert. we examine yearly averages at both the paragraph level and the opinion level. 11 we focus only on paragraphs that involve interpretation, and code paragraphs classified as "formal" with a 1 and paragraphs classified as "grand" with a 0. these yearly averages, therefore, reveal the proportion of interpretive paragraphs that classify as formal as opposed to grand. figure 3 plots the yearly averages over our series: the left panel (panel a.) shows the yearly average at the paragraph level, and the right panel (panel b.) aggregates paragraphs within documents to show the yearly average at the opinion level. broadly understood, the historical trends in our predictions converge with the views of llewellyn and other legal observers. that is, the period after the civil war and before world war i was characterized by formal judicial reasoning; the mid-century was characterized by grand legal reasoning; and we now live in a period of formalist resurgence. the story is essentially the same at the paragraph or the document level. but our predictions also allow for a more granular assessment of the historical periods. to illustrate this, we use dashed vertical lines in figure 3 to denote important historical events: in 1905, the supreme court decided lochner v. new york, foot_9 which some observers note as a highwater point for formalism; in 1937, a year of "judicial revolution," in which the supreme court is widely viewed to have shifted its jurisprudence from opposition to acthe same as trends in paragraphs. these results represent some of the first long-run quantitative characterization of trends in jurisprudential philosophies. they both broadly support the qualitative characterizations of legal scholars and provide opportunities for refinement of legal theory and historical accounts. ## conclusion we found that for a task involving abstract reasoning in addition to specialized domain-specific knowledge, it remains essential to have an annotated dataset created by domain experts. although other work has shown that generative models are able to replicate annotation for complex tasks using carefully crafted prompts, we demonstrate that models fine-tuned on a sizable dataset of expert annotations perform better than models instructed to perform the task through in-context and cot prompts. we recommend that researchers use caution when employing non-fine-tuned generative models to replicate complex tasks otherwise completed by humans or with human supervision. best practices would call for human validation of generative model results and an assessment of costperformance tradeoffs with respect to in-domain models. 13 this revolution is also known as the "switch in time that saved nine," referring to the changed voting behavior of justice owen roberts in response to the running threat by president roosevelt to pack the court.