E - NER — An Annotated Named Entity Recognition Corpus of Legal Text (2022)

Abstract: Identifying named entities such as a person, location or organization, in documents can highlight key information to readers. Training Named Entity Recognition (NER) models requires an annotated data set, which can be a time-consuming labour-intensive task. Nevertheless, there are publicly available NER data sets for general English. Recently there has been interest in developing NER for legal text. However, prior work and experimental results reported here indicate that there is a significant degradation in performance when NER methods trained on a general English data set are applied to legal text. We describe a publicly available legal NER data set, called E-NER, based on legal company filings available from the US Securities and Exchange Commission’s EDGAR data set. Training a number of different NER algorithms on the general English CoNLL-2003 corpus but testing on our test collection confirmed significant degradations in accuracy, as measured by the F1-score, of between 29.4% and 60.4%, compared to training and testing on the E-NER collection.

## introduction named entity recognition (ner) aims to identify names of specific objects in the world (mostly nouns with few exceptions), such as the name of a person, location and organization, which indicate possibly important phrases that readers should pay attention to. ner has been used in a variety of downstream tasks such as question answering @xcite , document deidentification @xcite , relation extraction @xcite , and machine translation @xcite . consequently, there has been considerable work on ner using general language corpora @xcite and a variety of test collections are available. previous work has examined domain-specific ner, e.g. in finance @xcite @xcite , biomedical @xcite , online user-generated content @xcite , and legal (luz de @xcite applications, and found that the performance of domainspecific ner systems was poor if trained on general language corpora. constructing test collections for specialist domains can be a time consuming task requiring manual annotation of a corpus. to reduce this effort there has been considerable recent interest in transfer learning, such as pre-trained language models @xcite . nevertheless, there remains a need for specialist test collections whether for training or fine-tuning. legal text is one specialist domain where ner is of interest, due to its usefulness in assisting other legal tasks such as record linkage @xcite , court case linkage @xcite , contract analysis @xcite , prediction of judicial decisions @xcite , credit risk assessment @xcite , and questionanswering systems @xcite . however, despite increasing interest in this sub-domain, there is no publicly available corpus for the evaluation of ner methods for legal applications. this paper describes e-ner, an annotated ner collection of legal documents, 1 based on publicly available legal company filings in the united states securities and exchange commissions' edgar database. overall, we deployed four ner models to compare classification performance when (i) trained and tested on general english, (ii) trained on general english and tested on e-ner, and (iii) trained and tested on e-ner. the results support insights from earlier work, i.e. we observed significant performance degradation when trained on general english but tested on legal text. our experiments justify the utility of a domain-specific (legal) ner corpus. ## related work the primary contribution of this paper is a legal-english test collection for ner. we do not propose a new algorithm for ner and consequently restrict our description of ner methods to those used in our experimental work. hidden markov models (hmm) @xcite ) can be used to label sequences. @xcite demonstrated the application of hmm to ner. conditional random fields (crf) @xcite is another sequence labelling model which improves on hmm, by relaxing the stationarity and the output independence assumptions. @xcite and @xcite demonstrated the application of crf to ner. in more recent years, pre-trained language models @xcite and prompt-based learning @xcite have demonstrated superior performance. bidirectional encoder representations from transformers (bert) @xcite is a pre-trained language model which is based on transformers @xcite . bert pre-trains on a large corpus of non annotated text, performing self-supervised tasks, namely masked word prediction and next sentence pairing. bert can facilitate transfer learning: the model parameters from the pre-training step are used during the fine-tuning step, in order for the model to learn downstream tasks such as ner @xcite @xcite . there exist publicly available annotated ner data sets for general english text, such as @xcite , wnut17 @xcite , and the wikipedia gold standard corpus @xcite , as well as for other languages @xcite @xcite . for legal domainspecific data sets, non annotated legal text is abundant, as detailed in @xcite . for example, the pre-training of legal-bert @xcite is performed on a corpus of non annotated documents consisting of legislation, court cases, and contracts from the uk, us, and the european union. however, the fine-tuning of legal-bert is based on an annotated data set contracts-ner that is not publicly available. 2021 ) created a data set consisting of header sections of court cases text (the header section will declare the parties involved in a court case). finally, we also note that the edgar database has been used by @xcite to create an annotated data set, called finer, which contains over 1.1 million sentences. however, this data set is tagged with extensive business reporting language (xbrl) tags, and it is used for numeric entity recognition. ## edgar-ner (e-ner) data set we first describe the source documents that constitute the edgar-ner (e-ner) data set. we then enumerate the named entity classes, which slightly extend those used by conll-2003 (conll), foot_1 which is widely used in the ner community. financial entities, such as companies, individuals, and funds, that are registered with the united states securities and exchange commission (us sec) are required by law to submit financial filings to the electronic data gathering, analysis, and retrieval system (edgar). all filings in the edgar data set are publicly available. there is a wide variety of different filings some of which contain almost no text, e.g. form 3 (initial statement of beneficial ownership of securities) or form 4 (statement of changes in beneficial ownership of securities). we have arbitrarily chosen the year 2010 and downloaded 52 documents. the 52 edgar documents consisted of a variety of different filings. we only selected filings that contain content in the form of sentences, such as form 10-q, which are company quarterly reports, or form 8-k, which are used by companies to announce major events relevant to their shareholders. the 52 documents consist of 24 different types of forms. please see appendix a for details. the filings were downloaded using the index file 3 provided by edgar, in the form of html text. each document was pre-processed using the python package "beautiful soup" to extract sentences. we remove: an illustration of what elements we removed or kept in an example filing is shown in appendix b. after preprocessing the 52 documents, we split the document into sentences by identifying the line breaks in the document, and using the sentence tokenizer from the python nltk package. in total, we identified 11,696 sentences that required annotation. annotation of the collection was performed by the first author. note that we did attempt to outsource the annotation to a commercial crowdsourcing platform. we provided instructions, including the definitions of the named entity classes and the tagging guidelines. each document was assigned to 3 crowd workers to independently label so as to ensure the correctness of the labels. however, we found that there were significant discrepancies in the labels provided. while we acknowledge that this variation may have been due to our instructions being poor, it is our opinion that the task has a significant difficulty for a non-expert. the conll-2003 data set defines 4 classes of named entities (and the class "outside" for nonnamed entities) 4 as enumerated in ## experiments to verify the need for a legal ner collection, we evaluated the performance of four ner methods by (i) training and testing on a general english collection (conll), (ii) training on general english, but testing on our legal collection (e-ner), and (iii) training and testing on our e-ner collection. the conll collection is subdivided into train, validation, and test partitions, as indicated in ## conclusions and future work this paper describes the publicly available e-ner data set, derived from company filings from the us sec edgar data set. the collection contains over 400,000 tokens, and as such, is of similar size to the conll-2003 collection. however, the number of ne phrases (almost 9,000) is only about 25% of the number of ne phrases in the conll corpus. in part, this reflects the statistical differences between general and legal english, where we observed that the sentence length for legal english (34.5 words) is much larger than for general english (13.7), and that the token length of a ne in legal text is longer (2.68 tokens compared to 1.45). in addition, the fact that e-ner encompasses only 52 documents from edgar might also contribute to this discrepancy. our experimental results compared the performance of four ner methods when trained and tested on combinations of general and legal english. our results reaffirm that for legal ner in-domain performance is significantly degraded when training without using specific in-domain data. there is a number of potential future research directions. first, there is a variety of legal specialities, e.g. finance, civil litigation, and criminal law. further work is needed to investigate how ner models perform in various legal sub-domains -how diverge and large should annotated corpora be for legal ner? to this end, we plan to create annotated datasets for other types of legal documents, such as court proceedings or contracts. in addition, the evaluation of ner models using state-of-the-art methods and language models in legal nlp might unveil more informative results and drive potential methodological improvements.