Ginn-Khamov at S em E val-2023 Task 6, Subtask B : Legal Named Entities Extraction for Heterogenous Documents (2023)

Abstract: This paper describes our submission to SemEval-2023 Task 6, Subtask B, a shared task on performing Named Entity Recognition in legal documents for specific legal entity types. Documents are divided into the preamble and judgement texts, and certain entity types should only be tagged in one of the two text sections. To address this challenge, our team proposes a token classification model that is augmented with information about the document type, which achieves greater performance than the non-augmented system.

## introduction the goal of the legal ner subtask of the legal-eval task is to perform named entity recognition (ner) on english legal texts @xcite . we are interested in detecting entities specific to the legal domain, such as case numbers, as well as differentiating between legal roles for people, such as lawyers and judges. this information can then be used for downstream tasks such as information retrieval. while industry-standard ner systems such as spacy's transition-based parser @xcite achieve state-of-the art performance on most standard ner tasks, it can be desirable to have a customizable system for a domain task @xcite . custom ner systems can support domain-specific entity types and learn to detect entities in structured domain-specific documents. our primary system utilizes a pretrained language model with a token classification head. the data provided by @xcite consists of annotated sentences from indian english court judgement documents, which are divided into two sections, the preamble text and judgement text. we observed that while entities of certain classes appear in both the preamble and judgement texts, they should only be identified in one of the two document classes. to address this imbalance, we introduced augmentations that aided our system in making accurate predictions for each document class. we discover that our system is able to achieve similar performance to the baseline on the validation data, and that certain augmentations allow the system to learn more quickly and accurately. our system shows improvements for certain entity types, although certain entity types remain difficult to label. our code is available at https://github.com/ michaelpginn/semeval2023-legalner . ## results the overall performance for each model on the validation data is listed in our best system achieved an f1 of 72.7 on the test data, ranking 12 of 17. as this performance is significantly worse than our validation set, this indicates that either our system is overfit to the validation data or the validation and test data have significantly different distributions. the former seems unlikely, since we only tested the models listed against the validation data, and all of them performed well on both the training data and unseen validation data. thus, we believe that the test data must have been different than the training and validation data in some way, such as having different distribution of entity types (maybe the patterns we observed with entities in preamble and judgement texts weren't consistent?). we do observe that the test data has a much greater proportion of sentences that were truncated by our model input length, which could be partly responsible for the error. ## conclusion in this paper, we describe a fine-tuned transformer model for named entity recognition in the legal domain, which fails to outperform the sota, although producing very similar results. using a custom transformer architecture is desirable in domainspecific tasks, as it allows customization to the task at hand. we demonstrate that in situations where documents should not all be labelled in the same manner, augmenting data using a trained classification model can improve performance. to this end, the simplest and most effective solution is to append a special token to the sequence indicating document class. we also demonstrate that domain-specific pretrained language models do not necessarily offer performance benefits over general-purpose models. future work could involve augmenting input data further using information such as word shape.