Legal Named Entity Recognition with Multi-Task Domain Adaptation (2022)

Abstract: Named Entity Recognition (NER) is a well-explored area from Information Retrieval and Natural Language Processing with an extensive research community. Despite that, few languages, such as English and German, are well-resourced, whereas many other languages, such as Romanian, have scarce resources, especially in domain-specific applications. In this work, we address the NER problem in the legal domain from both Romanian and German languages and evaluate the performance of our proposed method based on domain adaptation. We employ multi-task learning to jointly train a neural network on two legal and general domains and perform adaptation among them. The results show that domain adaptation increase performances by a small amount, under 1%, while considerable improvements are in the recall metric.

## introduction legal is one of the domains where ner plays a central role, especially in document processing, where it is used for identifying key elements like the court name, the name of the parties in a case, or the case number @xcite . in recent years, interest has grown in the research community for performing various tasks on legal documents, known as legalai @xcite . one direct application of these extracted named entities is document organization and search. however, they can be further incorporated into other systems like document anonymization, judgement prediction, or case summarization, offering additional insights to legal professionals @xcite . although still considered under-resourced, romanian is one of the languages that has seen a recent expansion with the introduction of two bidirectional encoder representation from transformers (berts) @xcite trained on romanian text @xcite , three named entity corpora @xcite @xcite , over three hundred hours of publicly-available transcribed speech @xcite , and a benchmark that tracks the progress of various romanian nlp tasks @xcite . in addition, domain adaptation research showed that we could perform knowledge transfer between datasets using effective methods in both supervised @xcite and unsupervised @xcite settings. in this work, we want to take advantage of these recent developments and explore the area of domain adaptation with a task discriminator on the romanian language. on a more granular level, we experiment with domain adaptation from the general to the legal domain, using the romanian named entity corpus (ronec) @xcite as a reference and romanian legal ner corpus (legalnero) @xcite as a target. our proposed neural architecture employs multiple components. a pre-trained bert layer generates the feature representation. long short-term memory (lstm) @xcite is utilized in bidirectional configuration to capture both left-to-right and right-to-left dependencies. conditional random fields (crfs) @xcite generate the predictions based on the conditional probability of the sequence. we employ multi-task learning @xcite to train the model on legal and general domains jointly, and on top of that, we apply domain adaptation as described by @xcite , but in a supervised setting. furthermore, to explore the robustness of our approach in another but better-resourced language, we apply the same methodology to german and investigate domain adaptation from germeval 2014 @xcite to german legal ner (german ler) @xcite . ultimately, we evaluate our approach through visualizations and analysis of the predictions. we summarize the contributions as follows: often, ner is jointly addressed along with relation extraction @xcite . while ner is usually solved using recurrent neural networks, relation extraction can be handled using convolutional @xcite and feed-forward layers @xcite @xcite . to generate labels for new types of entities and relation extraction in automated systems, distant supervision @xcite is utilized based on a dataset of entities. improvements rely on introducing a reinforcement learning module in the tagger that selects clean data for the model architecture during training @xcite . ner tasks can become challenging when entities are nested; often, they address flat ner. generally, classical approaches do not consider nesting and treat this task as dependency parsing @xcite . the method relies on embeddings generated via bert for word-based embeddings and convolutional layers for char-based embeddings. feed-forward layers and a biaffine model @xcite predict the entity spans. legal ner. previously, classical machine learning techniques such as support vector machines, naive bayes, and ontologies were utilized to detect named entities from legal documents @xcite @xcite . new datasets started to emerge in the legal domain since legal is one of the domains that received little attention @xcite . methods based on domain-specific embeddings and lstms combined with crfs were utilized in multiple languages, such as english @xcite , german @xcite , romanian @xcite , and portuguese (luz de @xcite . @xcite employed a two-learning-step approach that first trains a model on the ner task, which then creates features for training a second neural network model. this approach was evaluated on french legal documents, showing a significant reduction in the f1-score error. ## results this section presents results on baseline models and domain adaptation. we present the precision, recall, and f1-scores at the entity level (strict measures). in the end, we provide t-sne (van der @xcite visualizations of the embedding space on the feature space along with the limitations of the current approach. in appendix a, we present more detailed results. ## conclusions and future work we proposed a method based on multi-task domain adaptation in a cross-domain setting. the model architecture is based on contextualized word embeddings generated using bert, lstm, fully connected, and crf layers. we evaluated our approach on two languages (i.e., romanian and german) from two domains (i.e., general and legal). we observed minimal improvements in the german dataset while reducing performance on the romanian legal dataset. more research should be conducted in this direction. for future work, we strive to investigate the performance degradation further and analyze the effects of domain adaptation on the embedding space via t-sne visualizations. in addition, we want to evaluate a cross-lingual setting, considering the cross-language bert pre-trained model and performing domain adaptation between the same domain but different languages.