Detecting Arguments in CJEU Decisions on Fiscal State Aid (2022)

Abstract: The successful application of argument mining in the legal domain can dramatically impact many disciplines related to law. For this purpose, we present Demosthenes, a novel corpus for argument mining in legal documents, composed of 40 decisions of the Court of Justice of the European Union on matters of fiscal state aid. The annotation specifies three hierarchical levels of information: the argumentative elements, their types, and their argument schemes. In our experimental evaluation, we address 4 different classification tasks, combining advanced language models and traditional classifiers.

## introduction the study of argumentation in legal contexts is one of the most lively research areas at the intersection of artificial intelligence and law @xcite . it has its roots in logic, philosophy, and linguistics, as it studies how different claims and opinions are proposed, debated, and evaluated, considering their relations and inter-dependencies. the legal domain offers a natural scenario for the application of different argument models as well as novel machine learning and natural language processing techniques in order to perform legal reasoning @xcite @xcite , build specific ontologies @xcite , or support the teaching of law @xcite . argumentation is relevant to legal logic @xcite , case-based reasoning @xcite , the interpretation of judicial opinions and statutory laws @xcite @xcite , the summarization of judicial opinions @xcite . building tools capable of automatically detecting arguments in legal texts can produce a dramatic impact on many disciplines related to law, providing valuable instruments for the retrieval of legal arguments from large corpora, for the summarization and classification of legal texts, and for the development of ai systems supporting lawyers and judges, by suggesting relevant arguments and counterarguments. a crucial obstacle to providing effective automatic support to legal argumentation pertains to the knowledge bottleneck: legal arguments are only available in natural language texts, whose content has been so far only accessible with the help of domain experts. to overcome this limitation, recourse has been made to argument mining (am), i.e., the automated extraction of arguments from documents. am frameworks can be described as multi-stage pipeline systems, aimed at extracting natural language arguments and their relations from textual documents @xcite . each stage of the pipeline addresses a sub-task of the problem. a first stage usually consists of detecting which sentences in the input document(s) are argumentative, i.e., contain an argument or part thereof. once argumentative sentences are singled out, it is possible to detect the boundaries of the various argument components and their characteristics @xcite @xcite . finally, a last stage in the pipeline considers these components in order to predict the relationship between them and/or between the arguments they are part of @xcite . in this work, we contribute to this research domain by releasing demosthenes, a novel corpus of legal documents annotated for am. specifically, we focus on the first two stages of the pipeline in order to: (i) identify premises and conclusions; (ii) distinguish between legal and factual premises; (iii) identify argumentative schemes. additionally, we perform an experimental evaluation on all the tasks using multiple representations and classifiers. the paper is structured as follows. in section 2, we provide an overview of related work. section 3 describes the corpus we have created and the annotation procedure. section 4 concerns the experimental setting, while the results are presented in section 5. section 6 concludes. ## related work despite in the last decade the field of am has become a popular research area in natural language processing (nlp), there are yet limited studies focusing on legal texts and, in particular, on judicial decisions @xcite . among them, the targets of judicial am vary widely @xcite . some studies aim at extracting the arguments from generic unstructured documents @xcite ; others start from a given set of arguments and focus on aspects such as the identification of attack/support relations between them @xcite , or the classification of argument schemes @xcite . one of the main obstacles in providing effective automatic support to legal argumentation pertains to the knowledge bottleneck. like most interdisciplinary studies, creating and constructing annotated corpora is labour-intensive, as it is a complex and time-consuming task, requiring the guidance of legal experts, i.e., lawyers, being also familiar with legal arguments and the specific legal domain. indeed, a discrepancy exists between the way nlp researchers model and annotate arguments in court decisions and the way legal experts understand and analyze legal argumentation @xcite . in fact, under computational approaches, arguments are often treated as mere structures of premises and claims @xcite . in legal research, on the contrary, it it critical to also distinguish different kinds of arguments and classify them according to the rich typology that is rooted in the theory and practice of legal argumentation @xcite . finally, legal arguments may present themselves in different ways within different kinds of legal texts, depending on the on the domain of the law being addressed, and on the institutional position and legal culture of the authority that is producing such texts. unfortunately, there are a limited number of annotated corpora that fit the requirements just men-tioned, and they include a small amount of documents, withing specific areas of the law. thus the research community can highly benefit from the availability of new datasets, which as is the case of demosthenes, cover a sizable amount of examples, and include an attempt at classifying the identified arguments according to a legal typology. moreover, to the best of our knowledge, few works have analysed how natural-language argumentation is used in real courts @xcite . this situation leads to three urgent needs in legal am: (1) the creation of new annotated corpora, (2) possibly addressing different domains of the law; and (3) an analysis of how and to what extent models of arguments from legal theory can be reliably operationalized in terms of discourse annotations. the approach by poudyal et al. (2020) represents, to date, one of the few works whose goal was to implement a full-fledged argumentation mining system, specific to a single legal domain. mochales palau and moens (2011) created a corpus of 47 cases (judgments and decisions) from the open-source database of the european court of human rights (echr), in which they applied a sentence-level annotation scheme based on walton's model @xcite where each sentence was labeled as premise, conclusion or non-argumentative. more recently, poudyal et al. (2020); @xcite used the same guidelines to release a similar dataset of 42 documents. @xcite annotated judicial decisions selected from the u.s. court of federal claims also identifying sentences' inferential roles and support levels by using logical connectives to represent argumentative relations between premises and conclusions. @xcite published a dataset of judicial decisions from the board of veterans appeals. the decisions are annotated by legal experts with semantic information about arguments, including ten sentence roles and eight propositional connectives. the corpus initially contained 20 documents but was expanded subsequently @xcite . in this work, we aim to partially fill the mentioned gaps by: (1) creating a new annotated legal corpus; (2) focusing on a domain that is still unexplored in the field of legal argumentation, i.e., fiscal state aid; and (3) investigating whether argumentation schemes defined in legal theory, in particular by @xcite can be easily adapted to the cjeu reasoning, as made explicit in the court discourse. in particular, we focus on the detection of argumentative elements and their classification according to a hierarchical taxonomy of three layers, as detailed in the following. for what concerns the experimental part, previous works have addressed am in the legal domain using naive bayes and maximum entropy @xcite , factor graphs @xcite , and residual networks @xcite . more recently, advanced language models based on attention such as bert @xcite and roberta @xcite have been used and combined with lstms and cnns @xcite . in this work, we exploit a combination of advanced language models, namely sbert @xcite and legal-bert @xcite , and traditional classifiers. we used existing language models without fine-tuning them. this is in line with recent efforts in the nlp community toward efficient machine learning methodologies with limited computational footprint @xcite . ## corpus creation the source corpus consists of 40 decisions on fiscal state aids by the court of justice of the european union (cjeu), written in english. the decisions range from 2000 to 2018, i.e., since the cjeu's inception as a court of appeal in this domain. all documents have been downloaded from the eur-lex database and manually labelled. we have chosen this source since: (a) cjeu decisions usually contain a rich and diverse set of legal arguments (e.g., arguments appealing to statues, principles or precedents, according to different interpretive canons); (b) they have a standard (although not fixed) structure, in which argument chains are embedded and can be easily identified; (c) the selected decisions come from the same domain-i.e., fiscal state aids-which strongly relies on judicial interpretation; and (d) our annotators have some expertise in this domain. ## experimental setting in this study, we addressed four tasks. two are general argument mining tasks, namely argument detection and argument classification. the other two are rather domain specific and are type classification and scheme classification. they are defined as follows: we structured tc and sc as multi-label classification tasks since in both cases a single input sentence can have multiple labels. however, it is important to highlight that each sentence considered in these tasks has at least one label: there are no premises without a type, nor legal premises without a scheme. we did not enforce this constraint in our experiments and leave it for future work. for ad, as a first step, we pre-processed the documents removing periods from some common abbreviations (e.g., 'p.' for 'paragraph' and 'n.' for 'number'). the sentence segmentation was then performed based on periods, semicolons, and newlines. for all the tasks, we pre-processed the sentences by removing stop-words and punctuation symbols. experiments were conducted using 5-fold crossvalidation with folds determined at the document level, so that sentences of the same document belong to the same fold. the folds were created manually to balance their composition and guarantee that all scheme classes were represented in each fold. for all tasks we adopted three different representations of the input text: ## results and discussion for each task, we report the results obtained by each combination of embeddings and classifiers. we also report the performance of two simple baselines: a classifier that outputs a random value and one that always predicts the majority class. we measure the f1 score obtained for each class and their macro-average. ## conclusion we presented demosthenes, a new corpus for legal am in the fiscal state aid domain. the corpus consists in 40 decisions by the cjeu, which have been annotated on three hierarchical levels, identifying argumentative elements, their type, and argumentative scheme. we have defined 4 am tasks: ad, ac, tc, sc. our results highlight that legal-bert consistently obtains good scores in most settings and tasks. surprisingly, the tf-idf embeddings were often successful, suggesting that the lexical information may be informative enough to solve such tasks. for what concerns the classifiers, linear svc performed well in most of the settings. our results suggest that traditional classifiers are effective in many of the proposed tasks. we believe that these models can be considered strong baselines for further experiments involving state-of-the-art classifiers such as fine-tuned language models. in future work, we want to improve the scheme labelling by splitting the itpr class into multiple ones, and annotate the relationships between sentences. experimentally, we aim to implement oversampling and data augmentation techniques to overcome the strong unbalance of classes in each task. we also want to study the impact of pre-processing and the use of alternative classifiers such as logistic regression. finally, we want to improve the robustness of our experimental findings. for example, by considering multiple seed runs or applying the method proposed by @xcite .