P y E uro V oc: A Tool for Multilingual Legal Document Classification with E uro V oc Descriptors (2021)

Abstract: EuroVoc is a multilingual thesaurus that was built for organizing the legislative documentary of the European Union institutions. It contains thousands of categories at different levels of specificity and its descriptors are targeted by legal texts in almost thirty languages. In this work we propose a unified framework for EuroVoc classification on 22 languages by fine-tuning modern Transformer-based pretrained language models. We study extensively the performance of our trained models and show that they significantly improve the results obtained by a similar tool - JEX - on the same dataset. The code and the fine-tuned models were open sourced, together with a programmatic interface that eases the process of loading the weights of a trained model and of classifying a new document.

## introduction eurovoc 1 is a multilingual thesaurus which was originally built up specifically for processing the documentary information of the eu institutions. the covered fields are encompassing both european union and national points of view, with a certain emphasis on parliamentary activities. the current release 4.4 of eurovoc was published in december 2012 and includes 6,883 ids for thesaurus concepts (corresponding to the preferred terms), classified into 21 domains (top-level domains), further refined into 127 subdomains. additional forms of the preferred terms are also available and are assigned the same id, subdomains and top-level domains. multilingual eurovoc thesaurus descriptors are used by a large number of european parliaments and documentation centres to index their large document collections. the assigned descriptors are 1 https://data.europa.eu/data/datasets/ eurovoc then used to search and retrieve documents in the collection and to summarise the document contents for the users. as eurovoc descriptors exist in oneto-one translations in almost thirty languages, they can be displayed in a language other than the text language and give users cross-lingual access to the information contained in each document. one of the most successful recent approaches in document and text classification involves finetuning large pretrained language models on a specific task @xcite . thus, in this work we propose a tool for classifying legal documents with eurovoc descriptors that uses various flavours of bidirectional encoder from transformers (bert) @xcite , specific to each language. we evaluated the performance of our models for each individual language and show that our models obtain a significant improvement over a similar tool -jex @xcite . the the models were further integrated into the relate platform @xcite and an api was provided through the pythonpackage index (pypi) interface foot_0 that facilitates the classification of new documents. the code used to train and evaluate the models was also open-sourced foot_1 . the rest of the paper is structured as follows. section 2 presents other works in the direction of eurovoc classification. section 3 provides several statistics with regard to the corpus used to train and test the models in the tool. section 4 presents the approach used in fine-tuning the pretrained language models and the exact bert variants used for each language, together with a vocabulary statistics of the model's tokenizer on the legal dataset. section 5 outlines our evaluation setup and the results of our experiments, while section 6 presents the programmatic interface. finally, the paper is concluded in section 7. the algorithm used for classification was described in @xcite and it consists in producing a list of lemma frequencies from normalized text, and their weights, that are statistically related to each descriptor, entitled in the paper as associates or as topic signatures. then, to classify a new document, the algorithm picks the descriptors of the associates that are the most similar to the list of lemma frequencies of the new document. the initial release consisted of 22 pretrained classifiers, each corresponding to an official eu language. boella et al. @xcite , while focusing on the italian jrcacquis-it corpus, presents a technique for transforming multi-label data into mono-label that is able to maintain all the information as in @xcite , allowing the use of approaches like support vector machines (svm) @xcite for classification. their proposed method allows an f1 score of 58.32 (an increase of almost 8% compared to the jex score of 50.61 for the italian language). šarić et al. @xcite further explores svm approaches for classification of croatian legal documents and report an f1 score of 68.6. unfortunately, this is not directly comparable with the jex reported results since the training corpus is a different collection called nn13205. furthermore, the categories being used for the gold annotation represent an extended version of eurovoc for croatian, called crovoc. studies, such as in @xcite , have shown that neural word embeddings can store abundant semantic meanings and capture multi-aspect relations into a real-valued matrix, when trained on large unlabeled corpora using neural networks. considering a vocabulary v , an embeddings representation can be learned by means of a neural network resulting into an association of a real-valued vector w n of size n to each word. two neural network methods for automatically learning distributed representation of words from a large text corpus can be con-sidered: skip-gram and continuous bag of words (cbow) @xcite . in the case of cbow, a neural network is trained to predict the middle word given a context, while skip-gram uses a single word as input and tries to predict past and future words. bojanowski et al. @xcite introduced a method for runtime representation for unknown words by means of averaging pre-trained character n-grams, also known as subword information. bert has also been used to classify legal documents with eurovoc labels, with most of the work focusing on the english language. in @xcite , the authors studied the problem of large-scale multi-label text classification (lmtc) for few-and zero-shot learning and released a new dataset composed of 57k samples from eurolex on which several models were tested. the results showed that bert obtained superior performance in all but zero-shot classification. ## dataset statistics the training of bert models for the 22 languages was done using the same dataset that was used for training the jex models. the dataset is composed of two parallel corpora from the legal domain, jrc-acquis @xcite and the publications office of the european union (opoce), that were manually labeled with over 6,700 eurovoc descriptors identifiers (id). the eurovoc descriptors are hierarchically organised and can be converted into higher level microthesaurus labels (mt) and further into top-level domains (do). the number of documents in the dataset range from 17,858 documents in maltese to 41,989 documents in french. each document is labeled with multiple id descriptors, having an average of 6 id descriptors which can be equivalently converted to 5 mt descriptors or 4 do descriptors. in figure 1 we depict the distribution of the average number of id, mt and do descriptors per document, together with the difference between the minimum and the maximum number of documents per descriptor. the id, mt and do descriptors distributions are also highly unbalanced. figure 2 depicts the number of documents of the most frequent id, mt and do descriptors, organised in groups of 50, 5 and 1, respectively. each group contains the sum of the number of documents that are labeled with each descriptor in the respective group. as it can be observed, in each subplot, the number of documents that contain the descriptors from the first few groups is higher than the number of document that contain all the other descriptors. ## methodology the proposed approach for classifying the legal documents found in the two corpora is to fine-tune a pre-trained bert on each of the 22 languages. we follow the method introduced in @xcite where a simple feed-forward network with the weights @xmath0, e is embedding size of bert and m is the number of classes, and bias @xmath1. the sigmoid σ function is then applied to produce independent probability distributions ŷ over each 4 the mt and do descriptors are predicted by using a direct mapping scheme class: additionally, a dropout of 0.1 is applied on the feed-forward layer to regularize the model. the models are optimized by reducing the loss l computed as the average binary cross-entropy between the output probabilities ŷ and the target classes y, over the m classes (id descriptors).: because the flavours of berts vary from one language to another, the choice of the initial models for each language was made by using the following heuristic, based on the corpora used for pretraining: the vocabulary of bert plays an important role in the final performance of the model. broadly speaking, the fewer tokens each word is split into, the better the language model is expected to perform. in the legal documents in the corpus can be rather long and exceed the maximum number of tokens of 512 allowed by the bert models. to mitigate this, we simply keep only the first 512 in the document and discard the rest. this method has been shown to lead to approximately the same performance as considering the whole document @xcite . ## programmatic interface to ease the loading of models and the classification of documents, we created a programmatic interface in python that can be installed using pypi with the command pip install pyeurovoc. once the library is installed, a bert model is simply created by instantiating the class eurovocbert with one of the 22 languages. the class will either download the fine-tuned model from the repository or will use a local cached version of it. finally, the classification of a document is made by calling the instantiated model with the document text. more detailed information about the api and how custom pre-trained bert models can be finetuned on the dataset can be found at the source repository. an example of api usage is presented in appendix a. ## conclusion and future work document classification remains a relevant problem in nowadays society, aiding companies and government institutions to index their large textual database. this paper presented a tool for classifying legal documents with eurovoc descriptors that use various transformer-based language models, fine-tuned on the 22 languages that are found in jrc-acquis and opoce. we thoroughly evaluated the models on multiple splits of the data and the results showed that they significantly improve the performance obtained by another similar tool -jex. the pretrained models were made publicly available and they can be easily used to classify new documents using our api. one direction for possible future work is to improve the inference speed of the models by either distilling their knowledge in a smaller network @xcite or quantizing their weights @xcite . furthermore, we intend to include our results for legal document classification in language specific nlp benchmarks such as klej for polish @xcite , liro for romanian @xcite or evalita4elg for italian @xcite .