Named Entity Recognition in the R omanian Legal Domain (2021)

Abstract: Recognition of named entities present in text is an important step towards information extraction and natural language understanding. This work presents a named entity recognition system for the Romanian legal domain. The system makes use of the gold annotated LegalNERo corpus. Furthermore, the system combines multiple distributional representations of words, including word embeddings trained on a large legal domain corpus. All the resources, including the corpus, model and word embeddings are open sourced. Finally, the best system is available for direct usage in the RELATE platform.

## introduction natural language processing for the legal domain has its own unique challenges. this is due to the way legal documents are structured as well as to the domain-specific language being used. technology dealing with legal documents has received increased attention in recent years. this can be seen from the number of recent scientific papers being published, the existence of the natural legal language processing (nllp) workshop @xcite and different international projects dealing with natural language processing for the legal domain. named entity recognition (ner) is the process of identifying text spans that refer to real-world objects, such as organizations or persons, etc. one of the annotation schemes being used in a large number of works was introduced by the conll-2003 shared task on language independent ner @xcite and refers to names of persons, organizations and locations. this annotation scheme can be applied in the legal domain as well, thus allowing existing systems to try to annotate legal documents (with or without being adapted to legal text). however, domainspecific entities are usually added to enhance the in-formation extraction capabilities of text processing algorithms specifically designed for the legal domain. @xcite , while dealing with depositions, pleadings and trial-specific documents, propose including entities for attorneys, judges, courts and jurisdictions. @xcite proposed adding the entities date, money value, reference and "other" for analyzing legal contracts. @xcite proposed using 7 coarse-grained entity classes which can be further expanded into 19 fine-grained classes. in the context of the "multilingual resources for cef.at in the legal domain" (marcell) project foot_0 a large, clean, validated domain-specific corpus was created. it contains monolingual corpora extracted from national legislation (laws, decrees, regulations, etc.) of the seven involved countries, including romania @xcite . all seven corpora are aligned at topic level domains. the romanian corpus was preprocessed (split at sentence level, tokenized, lemmatized and annotated with pos tags) using tools developed in the research institute for artificial intelligence "mihai drȃgȃnescu", romanian academy (racai). named entities were identified using a general-purpose tool (pȃis , , 2019). this tool was designed for general romanian language and allowed only four entity types: organizations, locations, persons and time expressions. the tool was not trained on any legal texts. for the purposes of this work, we created a manually annotated corpus, comprising legal documents extracted from the larger marcell-ro corpus. we choose an annotation scheme covering 5 entity classes: person (per), location (loc), organization (org), time expressions (time) and legal document references (legal). references are introduced similar to the work of @xcite and the coarse-grained class proposed by @xcite , without additional sub-classes. thus, they are references to legal documents such as laws, ordinances, government decisions, etc. for the purposes of this work, in the romanian legal domain, we decided to explore only these coarsegrained classes, without any fine-grained entities. this has the advantage of allowing the corpus to be used together with other general-purpose ner corpora. furthermore, it allows us to judge the quality of the resulting ner system against existing systems. in order to train domain-specific ner systems, we constructed distributional representations of words (also known as word embeddings) based on the entire marcell corpus. finally, we explored several neural architectures and adapted them as needed to the legal domain. this paper is organized as follows: in section 2 we present related work, in section 3 is described the legalnero corpus, section 4 presents the legal domain word embeddings, section 5 describes the ner system architecture, while section 6 gives the results and finally conclusions are presented in section 7. ## related work legal ner is an important task in extracting key information from legal documents, such as dates, references to different types of legal documents, locations, organizations and persons. as zhong and tang (2020) stated, once the nes are identified and classified they can be used in workflows to perform different functionalities such as document anonymization or case summarization. one of the pioneering work in this research area was made by @xcite . the authors examined legal nes in us depositions, pleadings, case law and other legal documents using statistical models, context rules, and a lookup list of nes. they also developed different taggers for nes such as document type of jurisdiction, obtaining an f1 score of 0.92 for the nes belonging to jurisdiction class. this work formed the basis for @xcite to develop a tool for identifying, classifying and linking legal nes. they trained and evaluated different systems (stanford ner foot_1 , a support vector machine, and a neural network (nn)) on wikipedia and on decisions coming from the european court of human rights, obtaining an f1 score of 0.86 using nn and an f1 score of 0.56 using stanford ner. @xcite studied and evaluated on german legal data three ner systems. germaner, a generic german ner tagger, has been adapted to identify and classify nes such as persons, organizations, locations, and dates and money values. the second ner system used was dbpedia spotlight pipeline, an interlinking hub, a tool that can be used to perform annotation tasks on a text provided by a user @xcite . the third ner system employed in this task was based on contract templates to identify nes @xcite . for germaner pipeline and dbpedia spotlight pipeline the evaluation was performed on a corpus of 500 judgements, and achieved an f1 score of 0.8 and 0.87 respectively. the template ner system was evaluated on a corpus of contract templates and obtained an f1 score of 0.92. leitner et al. (2019) also evaluated two systems, based on conditional random fields (crfs) and bidirectional long-short term memory networks (bilstms), on ner for german language documents from the legal domain. the evaluation was performed on a german court decisions corpus annotated with 19 fine-grained classes, and also with 7 generalised coarse-grained classes. the best performance achieved by the crfs was 93.23 on fined-grained classes and 93.22 on coarse-grained classes, and the bilstms models reach a maximum of 95.46 for the fined-grained classes and 95.95 for coarse-grained ones. in the lynx project foot_2 (moreno-schneider et al., 2020) a set of services, including ner, were developed in order to help with the creation of a legal domain knowledge graph (legal knowledge graph -lkg) and its use for the semantic analysis of documents in the legal domain. @xcite described a method to generate contextual dictionaries for ner. the system was evaluated on a french legal corpus of 94 court decisions (276,705 tokens), which was annotated with 4 classes of entities. the best performance of this system was 96.52. even though there are several ner systems trained for romanian language both for general language (pȃis , , 2019) and specialised domains (mitrofan, 2019), regarding the legal domain, the experiments are very few and the performances are low (for example tufis , et al. (2020) note an average precision of 64.1% on a random sample extracted from the marcell-ro corpus, using the system devel-oped in (pȃis , , 2019)). apart from the new legal-nero corpus (see section 3), existing romanian ner corpora do not focus on the legal domain. romanian timebank @xcite is an annotated parallel corpus for temporal information. the ronec @xcite ) news corpus contains 26,377 named entities, belonging to 16 different classes. simonero (barbu mititelu and mitrofan, 2020) is a gold standard corpus for biomedical domain, manually annotated with four types of domain-specific named entities. ## the legalnero corpus annotation of the legalnero corpus was performed by 5 human annotators, supervised by two senior researchers at the institute for artificial intelligence "mihai drȃgȃnescu" of the romanian academy (racai). for annotation purposes we used the brat tool foot_3 @xcite , integrated in the relate platform @xcite . inside the legal reference class, we considered subentities of type organization and time. this allows for using the legalnero corpus in two scenarios: using all the 5 entity classes or using only the remaining general-purpose classes. the legalnero corpus contains a total of 370 documents from the larger marcell-ro corpus. these documents were split amongst the 5 annotators, with certain documents being annotated by multiple annotators. each annotator manually annotated 100 documents. the annotators were unaware of the overlap, which allowed us to compute an inter-annotator agreement. we used the cohen's kappa measure and obtained a value of 0.89, which we consider to be a good result. the raw annotations were obtained in the brat specific format, consisting of text spans, characterized by start and end positions with the associated entities. however, since many ner systems make use of token-based annotations, we further employed the romanian pipelines integrated in the relate platform (pȃis , , 2020) and annotated the corpus at token level. for tokenization, lemmatization, part-of-speech tagging and dependency parsing we used udpipe. finally, the named entity annotations were mapped to tokens and exported in conll-u plus format foot_4 , similar to the original format being used in the marcell-ro corpus. results presented in entity number person 914 location 2,276 organization 6,209 time 4,643 total 14,042 ## romanian legal-domain word embeddings previously computed romanian word embeddings (pȃis , and tufis , , 2018) made use of the representative corpus of contemporary romanian language (corola) @xcite . this large corpus contains texts from multiple domains, including the legal domain. however, in addition to the legal domain, the corola corpus contains texts from completely different areas, such as news, literature and mathematics. this makes the corola-based embeddings suitable for general tasks, but also allows for creation of legal-domain specific representations. for the purposes of this work, in addition to the already available corola embeddings, we constructed word representations based on the entire marcell-ro corpus. we used the same approach which obtained the best performing corola embeddings. thus, we used the fasttext 8 toolkit @xcite and produced vector representations of dimension 300, while considering only words appearing a minimum of 20 times. furthermore, the model made use of n-gram windows of dimension 5. the resulting embeddings are available for download within the relate 8 https://fasttext.cc/ platform foot_7 . for romanian language there are currently no legal domain contextual embeddings, like the legal-bert @xcite ) model for english. furthermore, existing romanian bert models, like @xcite ; s , tefan daniel dumitrescu et al., 2020), were not trained on corpora containing legal documents. ## system architecture our proposed ner model makes use of recurrent neural networks, based on bilstm cells, with a final crf layer. for features we considered word representations, character embeddings, gazetteer resources and known affixes. the word embeddings are initialized from a pre-trained model and are fine-tuned during training. the character embeddings are computed during training and the embedding layer is followed by a bilstm layer helping with the representation generation. for implementation we used a modified version @xcite of the neuroner @xcite software package. this implementation was further adapted to our needs in order to allow online model serving. the overall system diagram is presented in figure 1 . to construct the gazetteer resources we employed the geonames database for the country romania and the jrc-names foot_8 @xcite multilingual named entity collection. these two collections cover a large number of entity names like locations, organizations and persons. we trained multiple models using different features. this includes different word embeddings (corola and marcell embeddings) and also combinations of the two embeddings. previous work (pȃis , and mitrofan, 2021), @xcite has shown that using different word embeddings and combinations can improve ner performance. for each word representation we adapted the main bilstm layer size to match the embedding size. for example, in the case of corola embeddings we used a layer with size 300 and in the case of corola + marcell embeddings we used a layer with size 600. this was done to accommodate the increased vector size associated with the word representation. in addition to the main bilstm layer size, we used a character bilstm of size 25. furthermore, to prevent overfitting, a dropout of probability 50% was introduced. a gradient clipping @xcite with a value of 5 is used to deal with exploding gradients. finally, we use a stochastic gradient descent (sgd) algorithm with a learning rate of 0.005. besides experimenting with the aforementioned ner models, we also made tests with an ensemble method that combines the results from the different models. the idea was to see what combination provides better results and in what conditions. we've used four types of operations for combining the results. the first one is the union of two or more models. the second one is the full intersection of two or more models. in this case, if for the same set of tokens the annotations are different then none of them are represented in the final set. the third type is the implementation of a majority voting system, meaning that if an entity span has the same annotation in (n/2) + 1 cases, where n is the total number of models used, it is the winner. for example, given at least three models it takes at least two of the candidates with the same annotation to allow it to be represented in the final set. the fourth one is a merge between two or more models where for a given set of tokens the longest annotation between all of them is represented in the final set. afterwards, for each of the resulting final sets, the precision, recall and f1 scores are computed against the gold corpus for each entity types in the corpus. the macro average score is finally calculated. each of these pairs of combinations and scores are then recorded and a best that has the highest f1 score is calculated. the final step of measuring the precision, recall and f1 scores was done using neuroner evaluation script, in order to have a consistency with the initial models scores. ## results experiments were performed on the two perspectives associated with the corpus: the complete set of 5 classes (person, organization, location, time expressions and legal reference) and the reduced set (person, organization, location, time expression). in the second case, we took into consideration the additional annotations associated with the 4 remaining classes present inside the legal reference entities. furthermore, in order to explore the impact of different features, we conducted multiple experiments on each perspective. each experiment was allowed to train for at most 100 epochs. however, we used early stopping in case no improvement was perceived on the validation set for 10 epochs. as a result, neither of the experiments actually trained for the maximum of 100 epochs. the legalnero corpus was split into three subcorpora for training, validation during training and testing on data unseen during training. we used a 80% split for training and 10% for each of the validation and testing sub-corpora. the split was realized at file level, thus the training split contains 290 files, while the validation and test splits contain 40 files each. for reproducibility of the reported results as well as for comparison with future models, we offer the splits for download from the relate platform 11,12 . using the same splits, we also trained baseline models using well-known libraries such as spacy 13 . the spacy library provides a variety of tools for fast text processing and is developed as a modular pipeline. for implementation of our models we focused onto two critical components of the spacy pipeline, namely the components which are responsible for converting string tokens to vectors and the named entity recognition component. during training, besides word embeddings, we used spacy's class lexeme 14 , as an entry in the vocabulary. a lexeme has no string context, it is a word type, as opposed to a word token. it therefore has no part-ofspeech tag, dependency parse, or lemma (if lemmatization depends on the part-of-speech tag). after exhaustive tests, the representative model for each word embedding source also had its own optimal selection of the parameters that can be found at 15 . all of them made use of at least one of the affixes and some of them also used the shape. the spacy's training process, which is stopped automatically when the f1 score doesn't vary very much for a couple of epochs, guarantees saving the best model from all checkpoints. although it doesn't save any information during training, we used wandb 16 in order to retrieve the aforementioned data. using spacy, the best f1 score was 83.77, for all the five entity classes and using ro_core_news_lg 17 embeddings with a layer of size 300. when the legal class was excluded and with the same embeddings configuration, the f1 score obtained by spacy is 86.21. in the resulting best performing models are available for direct online usage through the relate platform 18 . this integration allows the user to enter a romanian legal document inside the platform's web interface, select the desired model, by using a ## conclusions this paper introduced a neural named entity recognition system designed specifically for the romanian legal domain. it employed the legalnero corpus for training and evaluation. the system is available for querying inside the relate platform and pre-trained models are available for download. as indicated in section 6, the best performing models made use of word embeddings trained on the legal-domain marcell corpus. when considering all the entity types available, corola and