Connecting Symbolic Statutory Reasoning with Legal Information Extraction (2023)

Abstract: Statutory reasoning is the task of determining whether a given law – a part of a statute – applies to a given legal case. Previous work has shown that structured, logical representations of laws and cases can be leveraged to solve statutory reasoning, including on the StAtutory Reasoning Assessment dataset (SARA), but rely on costly human translation into structured representations. Here, we investigate a form of legal information extraction atop the SARA cases, illustrating how the task can be done with high performance. Further, we show how the performance of downstream symbolic reasoning directly correlates with the quality of the information extraction.

## introduction statutory reasoning is the task of reasoning about legal cases with legal statutes. it is generally contrasted with case-based reasoning, and is complementary to it in legal systems based on common law @xcite . the statutory reasoning assessment dataset (sara) is a benchmark for statutory reasoning for us federal tax law @xcite . in some settings, laws and regulations can be encoded into first-order logic programs to reason about cases @xcite @xcite . in @xcite , sara statutes and cases were manually translated into prolog code, solving statutory reasoning in that context, in a process depicted in figure 1 . here, we take the standpoint advocated by @xcite , and start with the premise that a structured form of the statutes is available, while cases are stated in natural language. in practice, many governments across the world use expert systems to compute taxes owed by taxpayers, maintained by a dedicated team of experts @xcite . in contrast, translating cases into a cases payment_("paid"). patient_("paid","alice"). start_("paid","2017-12-31"). amount_("paid",33200). marriage_("married"). agent_("maried","alice"). … ?-s63_f_1_a @xcite . ## related work statutory reasoning while the present work is mainly concerned with ie, it is done for the purpose of statutory reasoning @xcite . this legal nlp task @xcite is close to that of legal entailment @xcite and case outcome prediction @xcite . large language models (llms) have recently set state-of-the-art on statutory reasoning @xcite . legal-domain ie ie has been done extensively on contracts, as a potentially high-impact application @xcite @xcite . the goal is to automate routine legal tasks, such as legal contract analysis @xcite and contract management @xcite . ie has also been attempted on regulatory texts: @xcite released a dataset of legal documents produced by the eu annotated with a set of legal terms. there are datasets for classifying paragraphs in policies @xcite , extracting subjectobject relationships @xcite , and identifying cross-references @xcite . closer to this paper, ie has been applied to case law. @xcite introduce a dataset of chinese court cases annotated with a custom event ontology. navas-loro and santos (2018) offer a review of event extraction in the legal domain, both annotation schemas and computational models, as well as annotations of european court of justice cases. @xcite provide a corpus of events based on the decisions of the european court of human rights, and compare statistical and rulebased models for event extraction. ## annotation of sara in @xcite and again in holzenberger and van durme (2021), the semantics of each sara case have been translated into a set of kbes. these annotations, based on a custom ontology in prolog, effectively make it possible to solve sara with a prolog program that reflects the semantics of the statutes. however, with ie in mind, the ontology has several design flaws. we will focus on two of them. first, the events are represented using strings and integers, without clear reference to the text of the case description. second, the use of the ontology is somewhat inconsistent. we improve over this annotation scheme to enable the training of ie models. foot_0 ## model overview we frame the problem of ie as mapping the case description (a string) to a tree of depth 2 or less, like the one on figure 3 . each node of the tree is a span in the case description, and each edge is labeled with an element from the ontology. the tree is built incrementally from the root: for each node, we predict its child nodes and the corresponding labelled edges. the example in figure 3 is predicted in 3 steps, predicting (1) the nodes with full lines, (2) the top 3 nodes with dashed lines, and (3) the bottom 3 nodes with dashed lines. this section details the steps in this procedure, and line numbers refer to algorithm 1, which describes how individual nodes and edges are predicted. algorithm 2 describes how trees are predicted. to put this approach into context: we use a span-based parser, which incrementally segments the input text and types each segment @xcite . we use the model described in section 4.5 of @xcite , which has state-of-the-art performance for framenet parsing @xcite . we make one major change: we use a multi-layer perceptron instead of a bil-stm. appendix c describes how predicted kbes are associated with probabilities. alice and bob got married on april 5th, 2012. alice and bob were legally separated under a decree of divorce on september 16th, 2017. x is mapped to features @xmath0, h 2 , ..., h n using the pre-trained encoder (line 2). given the span of the parent node, we augment h i with the distance between the span of the parent node and i. this distance is expressed in number of tokens, and represented using learned embeddings. if the parent node is the root of the tree, then we set the distance feature to the zero vector. we concatenate h i and the distance feature p i to obtain f @xmath1, p i ] (1 ≤ @xmath2., .] denotes concatenation (line 3). further, we compute x, the representation of the parent node, as the average of the tokens that represent the span of the parent node (line 4). if the parent node is the root, we set @xmath3. finding nodes given the parent node, we find its children as spans in the case description. we map each g @xmath4, x] (1 ≤ @xmath5. we use a conditional random field @xcite with bio tags to predict a set of spans @xmath6, s 2 , ..., s m }. we denote this bio tagger as t (line 5). ## conclusion we improve the sara ontology and re-annotate cases for their semantics. compared to the previous iteration of the sara dataset, the kbes and overall structure are now richer and more complete. we also take a step towards more consistent annotations, and more extractable kbes. we adapt state-of-the-art ie models to extract kbes from case descriptions. with empirical results on the sara dataset, we find that good performance with a strict metric can be reached on this small dataset, and that performance in ie translates directly into performance in statutory reasoning. we see three main directions for future work. first, as stated in section 5.1, the current ontology is not flexible enough to both (1) allow for kbes closely aligned to the case description, and (2) express complex relationships, in particular time relationships. this is one major limitation that could be addressed, and would likely require overhauling the expert system representing the statutes. second, we have so far assumed that the statutes were parsed into a logical form ahead of time. a meaningful step would be to augment the approach presented here with the automated extraction of the logical form of statutes, effectively performing semantic parsing. finally, cases in sara are not representative of the full scope and diversity of legal cases. real-world legal cases are generally much longer, the language is denser, and the phrasing is more diverse. we see the choice of an ontology and the annotation of data as two crucial challenges.