I talian Legislative Text Classification for Gazzetta Ufficiale (2023)

Abstract: This work introduces a novel, extensive annotated corpus for multi-label legislative text classification in Italian, based on legal acts from the Gazzetta Ufficiale, the official source of legislative information of the Italian state. The annotated dataset, which we released to the community, comprises over 363,000 titles of legislative acts, spanning over 30 years from 1988 until 2022. Moreover, we evaluate four models for text classification on the dataset, demonstrating how using only the acts’ titles can achieve top-level classification performance, with a micro F1-score of 0.87. Also, our analysis shows how Italian domain-adapted legal models do not outperform general-purpose models on the task. Models’ performance can be checked by users via a demonstrator system provided in support of this work.

## introduction the gazzetta ufficiale foot_0 (gu), in both its printed and digital editions, is the official source of the italian republic through which every legislative act issued by italian central and peripheral institutions, like the parliament, the constitutional court, the ministeries, the regional administrations, among others, is brought to the attention of citizens. this official journal plays a key role in italian lawmaking, as for any legislative measure to enter into effect, its publication in the gu is explicitly required by law. the istituto poligrafico e zecca dello stato foot_1 (ipzs), based in rome, is in charge of editing and publishing the gazzetta. a crucial step in the publication process is the assignment of each legislative measure to one or more labels from a subject index to concisely express its main semantic content and ease future search. however, so far this annotation has been carried out only manually, relying on a set of annotators that needed to be carefully trained to master around 1,600 labels available in the taxonomy. in order to support this task, reduce manual effort and ensure future consistency, we develop an automatic pipeline for the classification of gu documents. the task is particularly challenging due to the large number of labels, their different nature (i.e. mandatory vs. optional), the uneven representation of some categories and the different topics covered in the data. in our classification experiments we compare different transformer models for italian, focusing in particular on performance differences between general-purpose and legal adapted models. we release the classification models for reproducibility, which can also be tested through our online demo. 3 we also make available a github repository foot_3 with the annotated dataset used for our experiments, which consists of over 363k act titles from the general series, manually annotated by domain experts. ## related work in the legal domain, text classification has an established tradition, both in the monolingual @xcite and in the multi-lingual setting @xcite @xcite @xcite . moreover, the large availability of legal data, produced by national and supranational public institutions, set the stage for the development of domain-adapted models @xcite @xcite . as for italian, a multi-label classification system for bills has been proposed by de @xcite , based on bi-gru architecture using static word embeddings and employing a dataset of 28k legal document tagged with the teseo thesaurus. in our work, conversely, we compare the performance of different bert-based models @xcite for italian, including domain-adapted ones @xcite , on a large multi-label legal dataset covering 35 years of publication in the gazzetta ufficiale. ## gazzetta ufficiale subject index each italian legislative act entering the gu has to be manually assigned one or more thematic labels to classify its semantic content. such labels, provided by the gu subject index, are then used by the data administrators for classification (in the printed edition) and for indexing and retrieval of italian legislative acts (in the digital version). the manual labelling has been performed by a pool of expert annotators, trained by the data administration institution, over a time span of almost 35 years. layer n. of unique labels open labels (m) 781 closed labels (o) 1,019 references (o) 149 summaries (o) 71 ## dataset the publication of acts in the gu is structured in six series, depending on the regulatory body that issues the different types of acts: the serie generale (general series) and five special series. the general series includes all acts like ordinary laws, presidential decrees, ministerial decrees and resolutions, as well as other regulatory acts from the central and peripheral state administrations. to this respect, it represents the most relevant series in the gu and the backbone of the italian legislative process. the five special series, on the other hand, focus on specific institutions or types of acts: special series 1 contains judgements and orders issued by the constitutional court, special series 2 refers to regulations and directives of the european community, whereas series 3 includes regulatory and administrative acts issued by regional administrations. special series 4 and 5 are dedicated to the publication of documents relating to public exams and state contracts, respectively. while all acts in gu include a title and the proper body of the law, we focus in this work only on the classification of titles. indeed, if we consider the annotation procedure, in most cases annotators rely on the title of the act when assigning labels. only in cases where the title is not sufficiently expressive, or where the act includes several measures of different nature, will the annotator refer to the body of the act. also, although current large language models can deal with long documents @xcite , the use of titles alone allows us to minimize computational costs and avoid expensive preprocessing. in fact, by comparing different transformer-based models on the dataset, we intend to determine what the upper bound of performance is using only titles, which may be useful for future comparisons with document-level classifica-tion systems. our dataset contains 364,123 labeled titles of legislative acts published in the general series of the gu from 1988, when manual classification began, until early 2022. the temporal distribution of the documents in the dataset is depicted in figure 1 . according to the institutional annotation practices, each act (title) is manually annotated with at least one label chosen from the open labels set and, optionally, with one or more labels chosen among the open labels or from one of the other three layers (see ## legal text classification we cast the text classification task as a multi-class, multi-label problem, i.e. each target document (the act's title in this case) is to be labeled with one or more classes from the subject index. we compare different transformer-based models, after applying the same preprocessing for all settings. ## evaluation and error analysis performance results, computed on the test sets of the three folds, are reported in although not directly comparable due to the use of different datasets/annotation schemas, the performance of our models is in line with current stateof-the-art approaches to legal text classification @xcite , which however consider the full text of legal acts. in order to better understand the source of error in our models, we perform an extensive error analysis on the test sets used for evaluation. since in a multi-label setting it is not always possible to exactly map expected labels with predictions, due to possible many-to-many relations, we create a subset of wrong predictions where one-to-one correspondences are observed and use it as an approximation for the analysis. by manually inspecting the data, we observe that all models struggle with the same set of gold-predicted label mismatches. for example, only four pairs of labels are responsible, alone, for 10.5% of the overall error. it is worth noting that many of such mismatch types are related to pairs exhibiting strong semantic ties, like overlap (a), possible equivalence (b) subsumption (c, d). this pattern, enabled by the very structure of the subject index, is pervasive throughout the dataset and might be one of the reasons for a partially inconsistent use of the scheme over the years by different annotators, of which we have found more than one evidence in the gold data. at the other end of the distribution, we observe that 32% of the overall error is explained by a long tail of mismatches occurring 2 times or less. furthermore, in order to assess the model's ability to learn the hierarchical structure of the subject index, we analyze gold-prediction errors by mapping the respective labels' layers. as shown in ## conclusions this paper first introduces a novel annotated corpus of titles with manually assigned subject labels, which are part of gazzetta ufficiale, the official italian collection of legislative acts. we use this dataset to build and compare four multi-class classifiers for the legal domain, showing that specialised transformer-based models do not outperform general-purpose bert models for italian. in this first set of experiments we focused only on acts' titles, given that they yield promising results with a limited computational cost. however, in the future we plan to compare our results with those obtained by processing the full text of the acts. our analyses and experiments have highlighted some inconsistencies in the data, mainly due to the complex nature of the taxonomy and the large number of labels. as a next step, we will propose few improvements to make labels more consistent and speed up manual annotation. furthermore, it would be interesting to measure the gains in terms of time and effort when human annotators are given access to our system as a tool to support manual labelling.