Named Entity Recognition in Historic Legal Text: A Transformer and State Machine Ensemble Method (2021)

Abstract: Older legal texts are often scanned and digitized via Optical Character Recognition (OCR), which results in numerous errors. Although spelling and grammar checkers can correct much of the scanned text automatically, Named Entity Recognition (NER) is challenging, making correction of names difficult. To solve this, we developed an ensemble language model using a transformer neural network architecture combined with a finite state machine to extract names from English-language legal text. We use the US-based English language Harvard Caselaw Access Project for training and testing. Then, the extracted names are subjected to heuristic textual analysis to identify errors, make corrections, and quantify the extent of problems. With this system, we are able to extract most names, automatically correct numerous errors and identify potential mistakes that can later be reviewed for manual correction.

## introduction examining historical legal texts offers insight into the development of legal thinking and the practice of law. in order to facilitate computer processing, older legal texts are typically scanned from paper, microfilm or other physical media and then converted to text using optical character recognition (ocr), which introduces numerous errors. many of these errors can be corrected automatically using spelling and grammar correcting systems. however, the names of people, places and other proper names cannot be corrected easily, making the study of lawyers, judges and other people unreliable @xcite . one use of reliable names is inferring personal biases and connections that may affect outcomes @xcite . in order to address this problem, names need to be accurately identified in the text and then corrected and standardized in a process often called named entity disambiguation or ned @xcite . nonetheless, extracting accurate names is only part of the solution. in the future, organization names must also be extracted, and the respective roles must be identified. the process of computationally extracting names from text is more formally called named entity recognition (ner) and has been a difficult problem for many decades @xcite . furthermore, extracting names in legal text provides many domain-specific challenges @xcite . this paper describes a two-pronged approach for extracting the names of lawyers arguing cases: our system for extracting the names of lawyers in legal text uses a transformer-based neural network @xcite feeding a finite state machine. after extraction, the identified names are subjected to several heuristic rules to identify errors, misspelled names and name variations in order to attempt to uniquely identify the lawyers named. when errors cannot be corrected automatically, such as in names with alternative spellings, the extent of the errors is quantified. in order to develop, train and test this system, we used legal cases from the harvard caselaw access project (harvard university, 2018). this project includes the complete text of decisions from united states courts dating back to the 1700s, with over 40 million pages of text spanning over 360 years. in our analysis, we only focused on cases from 1900 to 2010 in jurisdictions that were states as of 1900. thus, alaska and hawaii were not considered. because states and courts often have different reporting styles that have varied substantially over the years, we segmented most of our analysis by state and then by decade. ## related work in a typical case text in the united states, lawyers are only identified in the header section on the first page using a relatively standardized format, usually called the "party names". they are rarely mentioned by name in the decision text. a typical party names text would read as follows: david p. sutton, asst. corp. counsel, with whom charles t. duncan, corp. counsel, hubert b. pair, principal asst. corp. counsel, and richard w. barton, asst. corp. counsel, were on the brief, for appellant. the text is usually a single complex sentence where all principal people, firms and their roles are identified in a mostly standardized and stylized format. because of the sentence's complexity, the text is sometimes difficult for non-lawyers and automated systems to decipher. the parsing problem is compounded because the style standards and norms vary by location and over time. in addition to containing spelling and transcription errors, words and names are sometimes given as initials, nicknames or abbreviations. all these types of things confound automated systems. thus, a solution to the problem can be divided into two parts: (i) extract the names from the text and (ii) standardize the name to identify individuals. one solution to the the first part of extracting names is documented by @xcite , who describe their work at westlaw (now part of thomson reuters) in 2000 identifying entities in us case law using bayesian modeling @xcite . their process extracts more than just names and involves parsing words in part by using a finite state machine that is specially tailored for each jurisdiction. more recently, @xcite propose a solution based on a neural network architecture that performs well across various domains, including legal text. in addition, @xcite have developed a very promising system to perform ner in german legal texts that was built and trained on their own dataset @xcite . this dataset was also used by @xcite in their ner work based on bert. these approaches apply generically to the entire legal text and are not focused on the grammatically challenging party names text. in any case, lack of a similar dataset for english prevents us from trying these approaches. our system differs from previous attempts in that it is an ensemble composed of a transformerbased neural network and a state machine rather than a single architecture. the state machine allows the inclusion of pre-established knowledge of the syntax and style of the named parties text, thereby increasing accuracy. this increased the accuracy by 10% compared with the state-of-the-art transformer-based flert model @xcite . ## historical vs. contemporary names contemporary texts are usually digitally encoded at creation and thus do not suffer from ocr-related errors. in addition, many anachronisms such as the practice of using just initials have been supplanted over time so that currently almost all lawyers use their full name and middle initial. abbreviations of names, such as "geo." for "george", have also fallen out of favor. this simplifies the identification of names in contemporary texts. in addition, contemporary names can be cross referenced with a standardized list of names such as westlaw, bloomberg, martindale-hubbell and similar directories that contain an almost comprehensive list of lawyers that can be used to uniquely identify individuals in the united states. however, such lists are subject to licensing restrictions and they have limited data for lawyers from the distant past. another consideration is that the same name can be used at different times and in different places to refer to different people. thus, a name is generally only unique for a specific time and place. a related problem is when the same name is used by a parent and child and only differentiated by the use of "sr.", "jr.", "ii", or similar suffix, if at all. yet another related problem is the use of an initial instead of a first name. this problem will be discussed in more detail in section 5.2. ## model architecture & experiments our source code is available at https:// harvard-almit.github.io/legal-nlp . to summarize, we developed a new system that combines a transformer neural network with a finite state machine. in addition we trained an existing architecture with a subset of the harvard caselaw access project. these two were compared with the benchmark flert model. ## performance analysis in this section, we evaluate the performance of our system regarding misspelled names and the use of initials instead of names and finally present a comparison analysis. ## conclusion we propose an ensemble model based on a transformer neural network architecture and a state machine for extracting names from party names in us case law text. the ensemble model improved the accuracy by approximately 10% from the flert model. however, once names were extracted, the problem of correcting for errors, misspellings and ambiguities could be fully automated. a number of techniques were discussed that help to quantify the extent of the problem and identify potential data quality issues. our analysis showed that the number of errors in some jurisdictions could exceed 15% in the decades before 1950, but that this number has been declining significantly over time.