Low-Resource Deontic Modality Classification in EU Legislation (2023)

Abstract: In law, it is important to distinguish between obligations, permissions, prohibitions, rights, and powers. These categories are called deontic modalities. This paper evaluates the performance of two deontic modality classification models, LEGAL-BERT and a Fusion model, in a low-resource setting. To create a generalized dataset for multi-class classification, we extracted random provisions from European Union (EU) legislation. By fine-tuning previously researched and published models, we evaluate their performance on our dataset against fusion models designed for low-resource text classification. We incorporate focal loss as an alternative for cross-entropy to tackle issues of class imbalance. The experiments indicate that the fusion model performs better for both balanced and imbalanced data with a macro F1-score of 0.61 for imbalanced data, 0.62 for balanced data, and 0.55 with focal loss for imbalanced data. When focusing on accuracy, our experiments indicate that the fusion model performs better with scores of 0.91 for imbalanced data, 0.78 for balanced data, and 0.90 for imbalanced data with focal loss.

## introduction obligations, permissions, prohibitions, rights, and/or powers are deontic modalities that commonly appear in legal documents. precise identification of deontic modalities within legal texts holds importance for understanding and determining legal positions, responsibilities, and legal actions. it can contribute to automated compliance checking @xcite @xcite , norm identification @xcite , and generation and analysis of legal policies and argumentation. deontic modality classification (dmc) allows machines to efficiently interpret regulatory texts and easily adapt to changes in legislation @xcite . thus, by pursuing the objectives of this research, we aim to address the limitations of traditional human or rule-based approaches and manual processing by leveraging neural networks for dmc. dmc is a natural language processing (nlp) task, in which, given a text with n normative statements @xmath0, s 2 , s 3 , ..., s n ), a classifier is trained to label deontic modalities in each statement @xcite . limited datasets are available for dmc. @xcite constructed a dataset based on the german tenancy law containing 913 sentences in german. the data protection regulation compliance (dapreco) @xcite dataset contains gdpr data represented in legalruleml @xcite . dapreco contains around 966 instances with 271 obligations, 76 permissions, and 619 constitutive rules. rather than using or constructing a dataset that focuses on a particular legal topic or domain (e.g., gdpr, tenant law), we aimed to enhance the generalizability of the results by randomly extracting 200 provisions from eu regulations. prior dmc research that relied on machine learning is commonly based on annotated datasets with little or no information about the annotation process and the agreement scores. as far as agreement scores are reported, scores of between .74 and .82 are reported, which is higher than observed by @xcite , who, after examining 29 publications on legal datasets, found average agreement scores of .76 (cohen's kappa), .675 (fleiss' kappa), and .677 (krippendorff's alpha), with the highest krippendorff's alpha value not exceeding .78. our research relies on an annotated dataset that resulted from multiple revision rounds by the annotators and an arbiter. we believe this approach yielded high-quality annotations, but with a limited number of annotated provisions. this is why we test fusion models, which are designed for low resource settings. with this research on dmc, we aim to answer the following questions -1. does a fusion model outperform a bertbased model? (rq1) 2. which strategy (imbalanced dataset, balanced dataset, focal loss) provides the best results for the bert-based and the fusion model? to answer these questions, we explore a model of contrasting design -a pretrained language model on eur-lex data called legal-bert -and a fusion model that is smaller, not pre-trained, and designed specifically for low-resource text classification. the models are experimented on a manually curated multi-class sentence level classification data, with the help of three annotators, preserving different interpretations of the law and the genuine disagreement regarding the deontic class. we report on the experiment setup, the various results, highlight the future opportunities for improving the models in order to give more insight on the challenges associated with low-resource dmc in eu legislation, and report the limitations of the proposed models. our study adds to previous studies by comparing different approaches, including fusion-models for low resource settings, using focal loss to address class imbalance, performing multi-class classification based on randomly selected statutory provisions from eu regulations, and building and testing models for annotators separately due to genuine disagreement amongst the annotators. ## methods the architecture and parameters of both legal-bert and the fusion model were optimized according to their performance on the data from one of the three annotators. focal loss was used to address significant class imbalance by providing more weight to difficult-to-classify examples and less weight to examples in the majority class(es). furthermore, the final fine-tuned versions of both models were trained separately on the three different datasets (one per annotator) for multi-class classification, where the full provisions were split into separate sentences or clauses for simplicity and where the possible labels were 'obligation','permission', 'power', 'prohibition', 'right' or 'none' (i.e., nonannotated text). ## discussion rq1: does a fusion model outperform a bertbased model? when evaluating with macro f1scores, it is seen that the fusion model outperforms legal-bert significantly. the similar results for the three annotators indicate that the models tested were adaptive to differences in input. overall, the findings suggest that fusion models might be a suitable approach for testing dmc in a low-resource setting. rq2: which strategy (imbalanced dataset, balanced dataset, focal loss) provides the best results for the bert-based and the fusion model? legal-bert performed with a macro f1-score of 0.45 for imbalanced data (with obligations having higher instances), 0.43 on balanced data, and fusion legal-bert predicted label true label predicted label true label none obligation none prohibition none prohibition obligation none obligation none obligation permission obligation none obligation permission obligation none obligation permission obligation right obligation permission obligation right obligation prohibition obligation right obligation prohibition obligation right obligation prohibition obligation prohibition obligation prohibition permission obligation obligation prohibition permission none obligation right permission power obligation right permission power obligation right prohibition permission obligation right permission none permission obligation permission power permission power as the data was skewed towards the 'obligation' class, evaluating with macro f1-score highlights the performance of the model per available class. the fusion model performed better with an increase of 10-20% with macro f1-scores across variations of the dataset when compared to legal-bert. legal-bert saw a decrease in performance when the data was more balanced, whereas the fusion model performed consistently well across dataset variations. it should be noted that the fusion model did not require any pre-training on large text corpora. moreover, the fusion model exhibited a lower discrepancy between the macro and weighted average accuracy, indicating its higher resilience to class imbalance and thus making it more suitable for lowresource settings. solely evaluating with accuracy can misrepresent the true performance of a model. ## conclusion and future work previous research has tested models that were trained on large corpus or fine-tuned on datasets of a single domain with data resources ranging from 500 to 1000 instances to perform dmc. with the annotation process being time-consuming, we relied on a carefully annotated dataset of 200 legislative provisions of eu regulations that were randomly sampled and that consequently covered a variety of legal topics. legal text can be interpreted in multiple ways, and instead of curating a single dataset based on inter-rater reliability (irr) scores, we trained multiple models for each annotator to understand the variation in model behavior. we tested previously researched bert-based models with low resources and evaluated them against a fusion model designed for low-resource text classification. legal-bert model and other variations through fine-tuning under performed for imbalanced data and further dropped in performance with balanced data despite the large corpora of eu legislative text utilized for pre-training. on the other hand, a fusion model ('cnn + a + bil-stm + cnn' variation) designed for low-resource text classification performed consistently well with both balanced and imbalanced data without the need for any pre-training on large corpora of text. future activities may focus on the challenging task of creating representative datasets that include more powers, prohibitions, and rights while maintaining a sufficient number of obligations and permissions. future research may also include exploring and testing approaches that handle differences in annotations between annotators as a result of genuine disagreement (as opposed to disagreements due to mistakes). in this respect, we plan to explore the ensemble approach for models trained on each annotator to take into consideration the disagreements between annotators.