Factoring Statutory Reasoning as Language Understanding Challenges (2021)

Abstract: Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that approached statutory reasoning as a monolithic textual entailment problem, with neural baselines performing nearly at-chance. To address this challenge, we decompose statutory reasoning into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog programs. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statutory reasoning are shown to benefit from the additional structure, improving on prior baselines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress.

## introduction as more data becomes available, natural language processing (nlp) techniques are increasingly being applied to the legal domain, including for the prediction of case outcomes @xcite @xcite . in the us, cases are decided based on previous case outcomes, but also on the legal statutes compiled in the us code. for our purposes, a case is a set of facts described in natural language, as in figure 1 , in blue. the us code is a set of documents called statutes, themselves decomposed into subsections. taken together, subsections can be viewed as a body of interdependent rules specified in natural language, prescribing how case outcomes are to be determined. statutory reasoning is the task of determining whether a given subsection of a statute applies to a given case, where both are expressed in natural language. subsections are implicitly framed as predicates, which may be true or false of a given case. @xcite introduced sara, a benchmark for the task of statutory reasoning, as well as two different approaches to solving this problem. first, a manually-crafted symbolic reasoner based on prolog is shown to perfectly solve the task, at the expense of experts writing the prolog code and translating the natural language case descriptions into prolog-understandable facts. the second approach is based on statistical machine learning models. while these models can be induced computationally, they perform poorly because the complexity of the task far surpasses the amount of training data available. we posit that statutory reasoning as presented to statistical models is underspecified, in that it was cast as recognizing textual entailment @xcite and linear regression. taking inspiration from the structure of prolog programs, we re-frame statutory reasoning as a sequence of four tasks, prompting us to introduce a novel extension of the sara dataset (section 2), referred to as sara v2. beyond improving the model's performance, as shown in section 3, the additional structure makes it more interpretable, and so more suitable for practical applications. we put our results in perspective in section 4 and review related work in section 5. ## sara v2 the symbolic solver requires experts translating the statutes and each new case's description into prolog. in contrast, a machine learning-based model has the potential to generalize to unseen cases and to changing legislation, a significant advantage for a practical application. in the following, we argue that legal statutes share features with the symbolic solver's first-order logic. we formalize this connection in a series of four challenge tasks, described in this section, and depicted in figure 1 . we hope they provide structure to the problem, and a more efficient inductive bias for machine learning algorithms. the annotations mentioned throughout the remainder of this section were developed by the authors, entirely by hand, with regular guidance from a legal scholar foot_0 . examples for each task are given in appendix a. statistics are shown in figure 2 and further detailed in appendix b. argument identification this first task, in conjunction with the second, aims to identify the arguments of the predicate that a given subsection represents. some terms in a subsection refer to something concrete, such as "the united states" or "april 24th, 2017". other terms can take a range of values depending on the case at hand, and act as placeholders. for example, in the top left box of figure 1 , the terms "a taxpayer" and "the taxable year" can take different values based on the context, while the terms "section 152" and "this paragraph" have concrete, immutable values. formally, given a sequence of tokens t 1 , ..., t n , the task is to return a set of start and end indices (s, e) ∈ {1, 2, ..., n} 2 where each pair represents a span. we borrow from the terminology of predicate argument alignment @xcite and call these placeholders arguments. the first task, which we call argument identification, is tagging which parts of a subsection denote such placeholders. we provide annotations for argument identification as character-level spans representing arguments. since each span is a pointer to the corresponding argument, we made each span the shortest meaningful phrase. figure 2(b) shows corpus statistics about placeholders. argument coreference some arguments detected in the previous task may appear multiple times within the same subsection. for instance, in the top left of figure 1 , the variable representing the taxpayer in §2(a)(1)(b) is referred to twice. we refer to the task of resolving this coreference problem at the level of the subsection as argument coreference. while this coreference can span across subsections, as is the case in figure 1 , we intentionally leave it to the next task. keeping the notation of the above paragraph, given a set of spans {(s i , e i )} s @xmath0, the task is to return a matrix @xmath1, 1} s×s where c i,@xmath2, e i ) and (s j , e j ) denote the same variable, 0 otherwise. corpus statistics about argument coreference can be found in figure 2 (a). after these first two tasks, we can extract a set of arguments for every subsection. in figure 1 , for §2(a)(1)(a), that would be {taxp, taxy, spouse, years}, as shown in the bottom left of figure 1 . structure extraction a prominent feature of legal statutes is the presence of references, implicit and explicit, to other parts of the statutes. resolving references and their logical connections, and passing arguments appropriately from one subsection to the other, are major steps in statutory reasoning. we refer to this as structure extraction. this mapping can be trivial, with the taxpayer and taxable year generally staying the same across subsections. some mappings are more involved, such as the taxpayer from §152(b)(1) becoming the dependent in §152(a). providing annotations for this task in general requires expert knowledge, as many references are implicit, and some must be resolved using guidance from treasury regulations. our approach contrasts with recent efforts in breaking down complex questions into atomic questions, with the possibility of referring to previous answers @xcite . statutes contain their own breakdown into atomic questions. in addition, our structure is interpretable by a prolog engine. we provide structure extraction annotations for sara in the style of horn clauses @xcite , using common logical operators, as shown in the bottom left of figure 1 . we also provide character offsets for the start and end of each subsection. argument identification and coreference, and structure extraction can be done with the statutes only. they correspond to extracting a shallow version of the symbolic solver of @xcite . ## baseline models we provide baselines for three tasks, omitting structure extraction because it is the one task with the highest return on human annotation effort foot_1 . in other words, if humans could annotate for any of these four tasks, structure extraction is where we posit their involvement would be the most worthwhile. further, @xcite have shown that the related task of semantic parsing of legal statutes is a difficult task, calling for a complex model. ## discussion our goal in providing the baselines of section 3 is to identify performance bottlenecks in the proposed sequence of tasks. argument identification poses a moderate challenge, with a language model-based approach achieving non-trivial f1 score. the simple parser-based method is not a sufficient solution, but with its high recall could serve as the backbone to a statistical method. argument coreference is a simpler task, with string matching perfectly resolving nearly 80% of the subsections. this is in line with the intuition that legal language is very explicit about disambiguating coreference. as reported in argument instantiation is by far the most challenging task, as the model needs strong natural language understanding capabilities. simple baselines can achieve accuracies above 50% for @truth, since for all numerical cases, @@xmath3. we receive a slight boost in binary accuracy from using the proposed paradigm, departing from previous results on this benchmark. as compared to the baseline, the models mostly lag behind for the dollar amount and numerical accuracies, which can be explained by the lack of a dedicated numerical solver, and sparse data. further, we have made a number of simplifying assumptions, which may be keeping the model from taking advantage of the structure information: arguments are instantiated in order of appearance, forbidding joint prediction; revisiting past predictions is disallowed, forcing the model to commit to wrong decisions made earlier; the depth of the dependency tree is capped at 3; and finally, information is being passed along the dependency tree in the form of argument values, as opposed to dense, high-dimensional vector representations. the latter limits both the flow of information and the learning signal. this could also explain why the use of dependencies is detrimental in some cases. future work would involve joint prediction @xcite , and more careful use of structure information. looking at the errors made by the best model in performance gains from silver data are noticeable and generally consistent, as can be seen by comparing brown and blue cells in in this work, each task uses the gold annotations from upstream tasks. ultimately, the goal is to pass the outputs of models from one task to the next. ## related work law-related nlp tasks have flourished in the past years, with applications including answering bar exam questions @xcite , information extraction @xcite @xcite , managing contracts @xcite @xcite and analyzing court decisions @xcite . case-based reasoning has been approached with expert systems @xcite @xcite , high-level handannotated features @xcite and transformer-based models @xcite . closest to our work is @xcite , where a dialog agent's task is to answer a user's question about a set of regulations. the task relies on a set of questions provided within the dataset. @xcite as well as preceding work @xcite tackle a similar problem in the science domain, with the goal of using the prescriptive knowledge from science textbooks to answer exam questions. the core of their model relies on several nlp and specialized reasoning techniques, with contextualized language models playing a major role. @xcite take the route of sorting questions into different types, and working on specialized solvers. in contrast, our approach is to treat each question identically, but to decompose the process of answering into a sequence of subtasks. the language of statutes is related to procedural language, which describes steps in a process. @xcite collect how-to instructions in a variety of domains, while @xcite focus on automotive repair instructions. @xcite exploit instructions in a game manual to improve an agent's performance. @xcite and @xcite turn to modeling textual descriptions of physical and biological mechanisms. @xcite propose models that generalize to new task descriptions. the tasks proposed in this work are germane to standard nlp tasks, such as named entity recog-nition @xcite , part-of-speech tagging @xcite , and coreference resolution @xcite . structure extraction is conceptually similar to syntactic @xcite and semantic parsing @xcite , which @xcite attempt for a subsection of tax law. argument instantiation is closest to the task of aligning predicate argument structures @xcite . we frame argument instantiation as iteratively completing a statement in natural language. chen et al. (2020) refine generic statements by copying strings from input text, with the goal of detecting events. @xcite extend transformer-based language models to permit inserting tokens anywhere in a sequence, thus allowing to modify an existing sequence. for argument instantiation, we make use of neural module networks @xcite , which are used in the visual @xcite and textual domains @xcite . in that context, arguments and their values can be thought of as the hints from @xcite . the prolog-based data augmentation is related to data augmentation for semantic parsing @xcite . ## conclusion solutions to tackle statutory reasoning may range from high-structure, high-human involvement expert systems, to less structured, largely selfsupervised language models. here, taking inspiration from prolog programs, we introduce a novel paradigm, by breaking statutory reasoning down into a sequence of tasks. each task can be annotated for with far less expertise than would be required to translate legal language into code, and comes with its own performance metrics. our contribution enables finer-grained scoring and debugging of models for statutory reasoning, which facilitates incremental progress and identification of performance bottlenecks. in addition, argument instantiation and explicit resolution of dependencies introduce further interpretability. this novel approach could possibly inform the design of models that reason with rules specified in natural language, for the domain of legal nlp and beyond.