Legal Terminology Extraction with the Termolator (2021)

Abstract: Domain-specific terminology is ubiquitous in legal documents. Despite potential utility in populating glossaries and ontologies or as arguments in information extraction and document classification tasks, there has been limited work done for legal terminology extraction. This paper describes some work to remedy this omission. In the described research, we make some modifications to the Termolator, a high-performing, open-source terminology extractor which has been tuned to scientific articles. Our changes are designed to improve the Termolator’s results when applied to United States Supreme Court decisions. Unaltered and using the recommended settings, the original Termolator provides a list of terminology with a precision of 23% and 25% for the categories of economic activity (development set) and criminal procedures (test set) respectively. These were the most frequently occurring broad issues in Washington University in St. Louis Database corpus, a database of Supreme Court decisions that have been manually classified by topic. Our contribution includes the introduction of several legal domain-specific filtration steps and changes to the web search relevance score; each incrementally improved precision culminating in a combined precision of 63% and 65%. We also evaluated the baseline version of the Termolator on more specific subcategories and on broad issues with fewer cases. Our results show that a narrowed scope as well as smaller document numbers significantly lower the precision. In both cases, the modifications to the Termolator improve precision.

## introduction automatic terminology extraction systems identify word sequences which are specific to a domain. the basic approach to terminology extraction involves syntactic processing to identify probable candidates. these are subsequently filtered using statistical techniques. what qualifies as key terminology varies depending on the field and intended purpose, resulting in differences in how terminology extraction is approached. this paper seeks to address the lack of terminology extraction approaches in the legal domain. @xcite specifically, we will tune an existing tool to extract terminology from various categories of us supreme court opinions. supreme court opinions are public domain and benefit from comprehensive tagging efforts and, as such, previous natural language processing work on supreme court case corpora has included automatic classification using word embeddings and neural networks (undavia et al., 2018). supreme court opinions, as compared with more technical legal documents, like contracts or academic papers, are written to justify decisions to a public audience and are thus written in a less constrained form and with more accessible language. nevertheless, since decisions are made on recurring legal topics, a regular vocabulary of issue-specific terms is likely to emerge. these are precisely the terms that we target and, in doing so, seek to contribute to the classification problem by providing an explicable identifying feature for categories of legal documents. we use washington university in st. louis database of supreme court decisions, manually annotated with broad and narrow categories for our experiments. this project uses the termolator (meyers et al., 2018), a high-performing open-source terminology extractor. the tool requires a foreground consisting of texts in the target topic and a multi-topic background both of which can be customized to need. the benefit of a curated background as opposed to a general corpus used by many other term extractors is that it intuitively allows for the extraction of the key terminology of a specific subdomain. for this paper, by specifying the foreground as a collection of supreme court cases in a given foreground topic and the background as opinions in a set of documents about other, varied topics, the terminology extracted are more likely to be specific to that foreground rather than also including general legal terms present in those cases that aren't necessarily representative of the foreground topic. the termolator also ranks the terms it finds based on a relevance score determined through web search and well-formedness metrics. the termolator was designed with a focus on science-oriented texts, so in this paper, we make several adjustments to the tool which improve its baseline precision for at least supreme court decisions. ## future work to build upon the proposed adjustments we made to the termolator, we aim to extend our data set to a larger size. the current data set contains 8400 supreme court cases with a significant variance in the number issue areas. having a more balanced number of cases in each broad issue could improve our results. a larger set in each category could mean that rarer key terminology is successfully identified, each potentially appearing with greater frequency in the foreground. similarly, a larger background set could improve results by providing more evidence that given phrases are not foreground-related terms. the difficulty in this lies in the lack of labelled cases; the data set used was manually labelled and likely incredibly labourintensive demanding a reading thousands of court cases. in tandem, the tasks can be mutually reinforcing, with a higher number of labelled cases, assuming high enough accuracy, allowing for larger data sets for key terminology extraction and, similarly, higher quality terminology can be used for greater accuracy in categorization. we are simultaneously looking into ways of automatically extending the manual classifications to more data. towards this end, we are currently working on extensions to the work published in (undavia et al., 2018). it is possible, that we could obtain better results if we used more data, even if some or all of it were automatically classified. note that of the approximately 64,000 supreme court cases, about half are unusable (really short) and 8400 are annotated. thus there are about usable 22,000 remaining. however, it is, possible, that other opinions, e.g., circuit court opinions could be used as well. ## conclusion our adjustments to the termolator have been shown to improve the precision of the output for supreme court cases. these adjustments involved, firstly, the filtering of the most frequently-identified errors followed by a more substantive change to the web-search contribution to the relevance score rankings. we further find that greater precision is achieved when the modified termolator is applied to broader-scope categories and issues with a greater number of cases. there is significant opportunity to improve and extend upon the investigation of this project to address the relative lack of research in the field of legal terminology extraction. the modifications to the termolator described in this paper will be released to the public via github.