NOMOS : Navigating Obligation Mining in Official Statutes (2023)

Abstract: The process of identifying obligations in a legal text is not a straightforward task, because not only are the documents long, but the sentences therein are long as well. As a result of long elements in the text, law is more difficult to interpret (Coupette et al., 2021). Moreover, the identification of obligations relies not only on the clarity and precision of the language used but also on the unique perspectives, experiences, and knowledge of the reader. In particular, this paper addresses the problem of identifyingobligations using machine and deep learning approaches showing a full comparison between both methodologies and proposing a new approach called NOMOS based on the combination of Positional Embeddings (PE) and Temporal Convolutional Networks (TCNs). Quantitative and qualitative experiments, conducted on legal regulations 1, demonstrate the effectiveness of the proposed approach.

## introduction in the legal context, obligations can be defined as "a legal constraint imposed by law and addressed to a juridical person" @xcite . they are often enforceable and backed by legal consequences in case of non-compliance establishing the rights and duties between parties, and regulating their interactions within a legal framework. in other words, an obligation represents a commitment to perform a specific action or refrain from certain behaviors. the interpretation of an obligation is not straightforward because it depends on several elements : the clarity of the text, the precision of the language, and moreover, the evaluation, the knowledge, and the experience of the reader. consequently, two or more individuals may perceive different obligations when presented with the same legal text. besides, even the most straightforward legal obligations are not always self-standing, and often encompass several sentences. they can also be fragmented on the paragraph level, whereby the paragraph begins with a generic phrase, followed by several sub-paragraphs, each of which introduces a different obligation that still relies on the initial generic phrase for correct interpretation. this sub-paragraph structure creates complications for traceability @xcite . in this paper, we present a comparison of several machine learning and deep learning methodologies for extracting obligations together with a solution based on the combination of positional embeddings (pe) and temporal convolutional networks (tcns). the contribution of this work is two-fold. 1. a full comparison between machine learning and deep learning methodologies on a set of different legislation containing a variety of obligations. the remainder of the paper is organized as follows. section 2 contains a description of the existing methods for extracting obligations in the legal domain. section 3 presents the methodologies for identifying obligations and the solution we adopted. quantitative and qualitative experimental results are shown in section 4. finally, conclusions are drawn in section 5. ## related work in the last years, the automatic analysis of legal texts has gained significant attention, and numerous studies and approaches have emerged that are capable of dealing with classifying complex legal texts. @xcite carried out experiments to classify legal clauses and sentences in the german civil code, legal texts in a different domain, and rental contracts, to investigate the portability of their models. they selected a taxonomy of 9 classes for classifying legal sentences, including for example duty, prohibition, definition, and consequence, the system being based on the function of the sentence for the subject of the law. the dataset consisted of 913 sentences that had been manually classified by legal professionals. they chose 6 types of models : multinomial naive bayes (mnb), linear regression (lr), support vector machine (svm), multilayer perceptron (p), random forests (rf), and an extra trees classifier (etc). the limitations of this study include the contract law focus -contract datasets being generally more restricted in scope and variation when compared to regulatory datasets -and the many semantic classes of legal sentences applied on a relatively small dataset. in de @xcite , the authors evaluated traditional machine learning methods, namely svms, against a knowledge-based method of standard phrases. their dataset was divided into 13 classes of sentences, including definitions, permissions, enactment dates, and obligations. while they were able to reach over 90% accuracy for recall and precision with the svm, the authors found that it did generalize well compared to the patternbased classifier on laws that were not included in the training set. this led them to conclude that the pattern-based classifier is the better choice because the classification criteria are transparent and customizable. it should be noted that only 181 out of 584 sentences were obligations. on the other hand, @xcite , trained a neural network classifier for legal obligations found in italian law. their classifier contained fewer categories (i.e. : no obligation, relevant obligation, and irrelevant obligation, based on their impact on financial institutions) and the dataset contained 10.628 clauses (the distribution of the categories within the training clauses is not provided). the labels for the training corpus were provided by non-expert annotators. their model of choice was the pre-trained umberto2 2 , to which they added a relu activation function for classification. the results showed an unreliable performance in distinguishing between relevant and non-relevant obligations, but significantly improved accuracy on the binary task of obligation versus no obligation, 2. https://huggingface.co/musixmatch/ umberto-wikipedia-uncased-v1 reaching an f-score of 0.97 for detecting not obligations and 0.91 for detecting obligations. however, the performance of the non-expert annotators in distinguishing between relevant and irrelevant obligations was also evaluated and found lacking. an additional limitation of this study was the absence of a validation dataset employed by the authors to evaluate the performance of their model. in @xcite , the authors performed a study on classifying sentences in financial regulations. in this study, the extraction of deontic modalities from regulatory texts was treated as a probabilistic rather than a logical problem. the training dataset was labeled by expert annotators and divided into 596 obligations, 607 permissions, and 94 prohibitions. for testing, the authors relied on a gold standard test set with separate documents of eu and uk financial laws. then, they used word2vec embeddings specific to the legal domain coupled with pre-trained google news embeddings as input to the classifiers. the authors compared a set of traditional machine learning algorithms, including lr and svm, to a set of deep learning algorithms such as lstm, cnn-lstm, and bilstm. when applying feature transformation and information gain (ig) feature selection on google news embeddings, svm outperformed all the other traditional methods on the test set but only reached an accuracy of 56.12%. on the other hand, the ann models produced significantly better results than svm, with two versions of bilstm reaching an accuracy of over 81%. however, the results are not aggregated per sentence classes across the board, making it difficult to know what the performance is when it comes to detecting obligations in particular. the use of temporal convolutional networks (tcns) in the legal domain is not yet widespread. in the state-of-the-art, there are just a few examples of the use of tcns for generic sentence classification tasks. for example, in @xcite , the authors described a bidirectional temporal convolutional network combined with an attention mechanism for sentence classification, reaching an accuracy rate of 91.47% on a public dataset. the main issue of such an approach is related to its effectiveness only in classifying short sentences like news. in this work, we propose a comparison of 3 stateof-the-art approaches, namely : tf-idf and svm, tcn, and bert. in our analysis, we focus only on tcn over lstm because of its ability to automatically extract features without the use of any convolutional layer, and moreover, because tcn possesses very long effective history sizes allowing the network to look very far into the past to make a prediction. we also present a novel approach based on the combination of positional embeddings and tcn that we adopted for detecting obligations in law texts. ## methodologies in sec. 2, we described the common approaches that can deal with obligation extraction. in our study, we selected and compared 3 methodologies, namely : 1) term frequency-inverse document frequency (t f -idf ) and support vector machine (svm), 2) temporal convolutional neural networks, and 3) bidirectional encoder representations from transformers (bert). moreover, we propose our solution for obligation classification. each methodology is detailed in the sections below. ## experimental results to evaluate all the approaches we described in sec. 3, a dataset of 74825 sentences has been labeled by expert lawyers and it is composed of 35850 obligations and 38975 not obligations. the sentences have been extracted from the code of federal regulations 3 and australasian legal information institute 4 . we divided the dataset according to the rule : 70% as the training set, 20% as the testing set, and 10% as the validation set. we used the same sentence splitter we applied for preprocessing the text in all the methodologies we proposed for this work to split the texts into sentences. for training and testing, we used a laptop running ubuntu 22.04 lts and equipped with an intel i9-12950hx cpu with 32 gb ram and an nvidia rtx a5500. we used tensorflow 5 for developing the deep learning algorithms, while we relied on scikit-learn 6 for the machine learning ones. ## conclusions in this work, we have presented a comparison of methodologies for extracting obligations from legal texts. apart from showing well-known methodologies, we proposed our own solution called nomos for dealing with such a classification problem based on the combination of positional and word embeddings and a temporal convolutional network. quantitative and qualitative results demonstrated the effectiveness of our approach in distinguishing obligations from non-obligation sentences reaching a f 1-score on both classes of 0.95. as future directions, we intend to introduce an attention layer to the network in order to focus on the subject, the verb, and the object of a sentence and distinguish between obligations directed at private parties and those directed at authorities.