L egal D iscourse: Interpreting When Laws Apply and To Whom (2024)

Abstract: While legal AI has made strides in recent years, it still struggles with basic legal concepts: _when_ does a law apply? _Who_ does it applies to? _What_ does it do? We take a _discourse_ approach to addressing these problems and introduce a novel taxonomy for span-and-relation parsing of legal texts. We create a dataset, _LegalDiscourse_ of 602 state-level law paragraphs consisting of 3,715 discourse spans and 1,671 relations. Our trained annotators have an agreement-rate \kappa>.8 , yet few-shot GPT3.5 performs poorly at span identification and relation classification. Although fine-tuning improves performance, GPT3.5 still lags far below human level. We demonstrate the usefulness of our schema by creating a web application with journalists. We collect over 100,000 laws for 52 U.S. states and territories using 20 scrapers we built, and apply our trained models to 6,000 laws using U.S. Census population numbers. We describe two journalistic outputs stemming from this application: (1) an investigation into the increase in liquor licenses following population growth and (2) a decrease in applicable laws under different under-count projections.

## introduction ai practitioners have long explored how to use automation to interpret the law 1 @xcite . recent advances in nlp and information retrieval have already enabled practical applications @xcite , such as legal question answering bots 2 , contract generation 3 , and automatic motion-filing @xcite . and, the legal reasoning capabilities of large language models (llms) are promising @xcite ) -gpt4 has been demonstrated to @xcite pass the bar exam. ...in counties having a metropolitan form of government and in counties having a population of not less than three hundred thirty-five thousand (335,000) nor more than three hundred thirty-six thousand (336,000), according to the 1990 federal census or any subsequent federal census, the magistrate or magistrates shall be selected and appointed by and serve at the pleasure of the trial court judge... figure 1 : paragraph from a sample law, tennessee § 36-5-402. the colored blocks represent the following legal discourse elements from our schema: probe, test, subject, consequence, object (see section 2). we train llms to identify these spans and build a web application to aggregate these span tags across state-level laws. however, fundamental challenges remain. as noted by @xcite , gpt3 models fail when confronted with simple, yet ambiguous conditions (or "tests") present in legal rules @xcite , a challenge documented in other models as well @xcite . additionally, the majority of legal study has been focused a few specific domains, like contracts @xcite , privacy policy @xcite , and corporate law @xcite , and the kinds of tasks heretofore studied have been highly domain specific 4 . benchmarks like @xcite are dominated by these use-cases, limiting our ability to get a general assessment of a model's abilities. it also limits our confidence about models' reasoning in understudied legal domains which are important to policy makers, journalists and academics, like state-level administrative law (). we see the need to introduce a unified mode of study that can quickly incorporate new areas and applications of law. in this work, we develop a uniform discourse schema for characterising a legal text. discourse analyses, or the study of functional role of text and its relations within in a document @xcite , has been successfully applied to areas like argumentation @xcite , dialogue @xcite and journalism @xcite . in journalism, for instance, choubey et al. (2020) use a unified discourse schema to describe textual relations between diffuse domains of journalism. in this work, we develop a legal discourse schema to address this need, which we apply to state-level legal texts. at the core, our schema seeks to answer the following key questions: (1) when does a law apply? (2) what are its consequences? (3) who is affected? we show that llms struggle to model this schema, yet it is useful for human practitioners. in sum, this paper makes three key contributions: we scrape over 100,000 law documents from 52 u.s. states and territories with 20 web-scrapers we build. u.s. law is always public domain yet, in practice, many states contract websites to host their laws and these websites use techniques to prevent bulk gathering legal text. we design our scrapers to be robust and to overcome these uncivil attempts to block consumption of public domain texts. we outline our discourse schema and modeling in section 2. next, we discuss our dataset collection process, including the web-scrapers we release for gathering public-domain u.s. state law text (section 3.1). in section 3.2 we describe our lightweight and modular span and relation annotation interface which we used to collect data. next, in section 6, we describe our webapp, where we surface our model's output to journalists and engage volunteers to improve our annotations. finally, we discuss an ongoing use-case to illustrate how one might use our app in section 6.1.1. ## a legal discourse schema a legal rule is a hypothetical imperative @xcite , or a conditional consequence. reasoning about these rules requires practitioners to understand how and whether conditions of the law are met; what the consequences are @xcite ; and who is affected by these consequences. as shown in figure 2 , modeling the different components of a legal doctrine as discourse units and how they interact as relations can be an effective way of discern meaning @xcite . identifying these parts poses a basic test of a model's legal reasoning and can also lead to practical use-cases (as @xcite showed in the journalism domain). we introduce the key parts in our schema, starting with span annotations and then relations. ## dataset creation in this section, we describe how we operationalized the schema discussed in section 2. we scrape a dataset of all state-level laws from 52 u.s. states and territories, which we discuss in section 3.1. we then sample a set of paragraphs to annotate. we build an annotation framework, described in section 3.2, and enlist four annotators, who collectively annotate 602 law paragraphs. ## legal entity and relational modeling we frame a new task using the data we collect: legal entity and relational modeling, or extracting legally significant spans and their relations. this task is analogous to end-to-end relation extraction (ere) @xcite . we will first describe two subtasks that traditionally compose ere, and how legal discourse can be modeled in this framework, then we will discuss methods, with a particular focus on how we can use this setup to interrogate the reasoning capabilities of large language models. ## results and discussion span-level tagging : our broader finding, though, is that this remains a challenging task. although our task dataset, at 400 documents, is small relative to other language resources, the spans in our schema are syntactically low-level. the spans divide relatively well into different parts of speech, like noun phrases and verb phrases; identifying such chunks in text has long been within the capability of even classical language models (sang and @xcite . future work either fine-tuning on other resources, or using law-specific models, might show improvements in these areas. relation identification and classification the relation identification task also shows a clear different between the baseline models, which we do not observe in the span-level tagging task. one explanation for the especially poor performance of asp @xcite is that the jointly learned model requires the model to make use of more data to fully learn the embedding layers. in fact, tasks that asp performs well on, like ace2005 @xcite , have 10x more documents and annotation than our dataset. we show more details in appendix e, figure 8 . ## practical use case: census 2020 to get feedback on our work from a preliminary group of users, we apply our models to a domain of statelevel law pertinent to journalists. in 2020, the u.s. census count faced multiple challenges, notably the trump administration's attempt to add the question: "are you a legal citizen?". many researchers hypothesized that populations, especially minorities, might be inaccurately counted (naylor, 2020; mervis, 2019; berry-james et al., 2020). scant insight existed, especially on the state-level, into how population counts were being used in law foot_14 : the corpus of state-level laws was too large and varied for journalists to parse. on the other hand, this provided an interesting case for discourse-based reasoning. population counts typically get used as a relatively unambiguous test. for example, see figure 1 , e.g. "in counties with less than 20,000, adjudicators shall..". our discourse models help us identify this occurring, and then we can develop ways to parse out the specific ways population is in test discourse. we describe the website we built to facilitate different explorations, and then we describe two such explorations that we received permission from the journalists collaborating with us to write about. we will focus on our own contributions in these collaborations. ## related work although the field of ai-driven legal aids is multifaceted and growing (kauffman and soares, 2020), free and open-source frameworks remain few @xcite @xcite . our discourse-driven web application, designed for legal exploratory analysis is one of the few ai-powered, free applications that exist, and the first to open source tools for legal document collection. for-profit legal inquiry systems, as mentioned above, are numerous. bloomberg law foot_18 , westlaw foot_19 , lexis-nexis 24 and wolters kluwar foot_21 are the four main services for legal research @xcite , which provide subscription-based, google-style searches. casetext foot_22 and ravel foot_23 were two upstart case-text search engines (although both have now been aquired); casetext offered crowdsourced annotations and ravel linked cases together to create visual maps of important cases @xcite . we similarly provide a way of collecting user-annotations, and a novel way linking together cases, although ours takes a discourse approach rather than an unsupervised clustering approach. various discourse schemas have been developed to understand law texts, including deontological logic-based schemas @xcite , and subject matter-specific schemas @xcite . ours is the first discourse-based approach to take steps towards a big-data approach by setting up a framework for the ingestion of crowdsourced annotations. finally, outside of the legal domain, other areas have experienced a growth in academically-oriented systems for human-in-the-loop inquiry. the covid-19 pandemic has produced a burst in nlp-driven corporacollection (wang et al., 2020), demonstrations @xcite @xcite and workshops @xcite . such concerted effort in the nlp domain to expose resources and build open tools for subject matter experts is an inspiring guide for how nlp researchers can contribute to wider inquiries. we hope such efforts expand to other domains as well, forming a common alliance between academics, civil-minded journalists and other researchers and end-users. ## conclusion we have sought to take steps towards a semantic understanding of legal texts, a goal long held in computational law @xcite . we show that large language models, while achieving impressive results in some parts of our task, show surprisingly weak performance compared to human annotators in others. language models have an important role to play in interpreting law and lowering the barrier of access to legal systems for citizens, journalists and academics. our task is an important step towards assessing a sturdy foundation and opening the door to more intensive legal tasks be considered @xcite . in this work, we have additionally presented three open-source components. (1) a web-app exposing a novel discourse schema and its application to state law referencing u.s. census counts. (2) a flexible and modular annotation framework that can be seamlessly embedded into web-apps to allow visitors to contribute and update annotations. (3) a set of web-scrapers to help researchers gather public-domain legal text. we demonstrated concrete utility to facilitate journalistic exploration with our discourse schema. our longer-term goal is to collect feedback and data, and improve our database and machine learning systems. we hope that such efforts can continue to push legaltech @xcite into a more open and accessible domain, and make it easier to understand the laws governing our society.