Legal_try at S em E val-2023 Task 6: Voting Heterogeneous Models for Entities identification in Legal Documents (2023)

Abstract: Named Entity Recognition (NER) is a subtask of Natural Language Processing (NLP) that involves identifying and categorizing named entities. The result annotation makes unstructured natural texts applicable for other NLP tasks, including information retrieval, question answering, and machine translation. NER is also essential in legal as an initial stage in extracting relevant entities. However, legal texts contain domain-specific named entities, such as applicants, defendants, courts, statutes, and articles. The latter makes standard named entity recognizers incompatible with legal documents. This paper proposes an approach combining multiple models’ results via a voting mechanism for unique entity identification in legal texts. This endeavor focuses on extracting legal named entities, and the specific assignment (task B) is to create a legal NER system for unique entity annotation in legal documents. The results of our experiments and system implementation are published in https://github.com/SuperEDG/Legal_Project .

## introduction and related work the significant growth of legal texts, particularly in highly populated countries, leads to a considerable burden on the workload of competent judges. automated text structurization is the first and essential step for other downstream tasks that may support judges' work. the distinct nature of the indian legal process and the terminology employed in legal texts are primary reasons why the semeval-2023 competition @xcite encourages research in this area @xcite . initially, researchers focused on rule-based methods, such as ralph @xcite , which depended on manually created rules and patterns to identify named entities in text. machine learning approaches were subsequently employed, with the hidden markov model (hmm) @xcite as one of the earliest methods modeling the probability distribution of named entity labels given a sequence of words. it laid the foundation for further research, culminating in the development of neural networks for ner, including the application of long short-term memory (lstm) @xcite , bidirectional lstm @xcite , convolutional neural networks (cnn), and iterated dilated cnn (idcnn) @xcite . the recent emergence of transformers @xcite has significantly impacted the nlp field, with promising results from ner models based on transformers. numerous research areas are active in ner, including methods combining bert, crf, and bilstm @xcite . these models can be trained on labeled ner datasets to recognize named entities in new texts. researchers @xcite investigate the application of bert models in the legal domain, proposing adaptation strategies, introducing a new dataset comprising over 62,000 legislative documents and 4,000 subject matter labels, and demonstrating the potential of pre-trained language models for large-scale multi-label text classification in legal document categorization. consequently, this paper's main contribution is the adoption of recent advances in heterogeneous models with a voting mechanism for ensembled entity annotation in legal texts. ## system overview figure 1 presents the architecture of the proposed system, which employs a combination of four heterogeneous models (section 3) and selects results through a voting mechanism. the system setup process consists of the following steps: 1) creating four initial models; 2) training each model. the recall and precision metrics are considered to evaluate the performance of these models. the model that demonstrates the best performance based on these metrics is subsequently chosen for prediction. the heterogeneous model results are combined using a textual voting method to derive the final prediction. ultimately, the voting process entails numerically polling labels from the four models and consolidating them into textual results. ## heterogeneous models in this paper, we consider heterogeneous models as three-tier designed ner annotators (see figure 2 ). we consider bert-based models in the first layer for learning and providing the contextual representation: albert @xcite and roberta @xcite . the second layer incorporates bilstm/idcnn to model the context. the third layer adopts crf to constrain predicted labels to ensure validity. sections 3.1 and 3.2 cover each layer's contents in greater detail. ## dataset in this paper, we utilize the indian judicial decisions dataset @xcite for model fine-tuning and experimentation. named entities are extracted from both the preamble and the main text of the judgment. the preamble of a judgment consists of formatted metadata, such as the names of the parties involved, the presiding judge, the representing lawyers, the date, and the court. a representative example is provided in fig. 4 , which contains four named entities: the petitioner (highlighted in yellow), the respondent (highlighted in green), the judge (highlighted in pink), and the lawyer (highlighted in red). the training dataset includes 9,435 judgments and 1,560 preamble records, while the validation dataset contains 949 judgments and 125 preamble records. in total, the dataset comprises 14 distinct named entity types. the named entity counts are detailed in hyper-parameter setting all considered language models are of the base size @xcite . we configure the training process to span 50 epochs. firstly, we restrict sentence lengths to a maximum of 500 tokens, as 90% of the sentences in the dataset are within this range, ensuring that the training results are not significantly impacted. sentences exceeding 500 tokens are split into smaller segments with lengths under 500 tokens, allowing for batched training. we utilize a batch size of 4 for this process. throughout the training period, we apply a dropout rate of 0.1 and a learning rate of 10 -5 . ## conclusion in this paper, we present a system for named entity annotation tailored explicitly for annotating objects 1285 in legal texts within indian court judgments. the system employs a voting mechanism over a threetier, heterogeneous ner model. we propose and utilize a diverse combination of model components, including bert-based language models for input representation, variations of bilstm, and idcnn encoders, all combined with a crf module for hidden representation. to assess the performance of our proposed system, we conduct experiments using the indian judicial decisions dataset provided by the competition organizers.