On Breadth Alone: Improving the Precision of Terminology Extraction Systems on Patent Corpora (2022)

Abstract: Automatic Terminology Extraction (ATE) methods are a class of linguistic, statistical, machine learning or hybrid techniques for identifying terminology in a set of documents. Most modern ATE methods use a statistical measure of how important or characteristic a potential term is to a foreground corpus by using a second background corpus as a baseline. While many variables with ATE methods have been carefully evaluated and tuned in the literature, the effects of choosing a particular background corpus over another are not obvious. In this paper, we propose a methodology that allows us to adjust the relative breadth of the foreground and background corpora in patent documents by taking advantage of the Cooperative Patent Classification (CPC) scheme. Our results show that for every foreground corpus, the broadest background corpus gave the worst performance, in the worst case that difference is 17%. Similarly, the least broad background corpus gave suboptimal performance in all three experiments. We also demonstrate qualitative differences between background corpora – narrower background corpora tend towards more technical output. We expect our results to generalize to terminology extraction for other legal and technical documents and, generally, to the foreground/background approach to ATE.

## introduction terminology extraction is the process by which specialized and domain-specific words and phrases are extracted from a set of documents. these techniques are actively used and researched to identify trends in technical documents, create domain glossaries, and improve the readability of technical documents, among their many uses. automatic terminology extraction (ate) methods are the class of linguistic, statistical, machine learning, or hybrid techniques designed to identify terminology in a specialized set of documents. ate now covers a broad class of methods that are in real use today and continues to receive research attention. most modern ate methods take advantage of a statistical measure of how important or characteristic a potential term is to a foreground corpus by using a second background corpus as a baseline. systems that use these statistics rely on an assumption that the foreground corpus is specialized and the background corpus is less specialized. the statistical methods can then use relative frequencies in the less specialized corpus and compare them to the specialized corpus -if a term is significantly more common in the specialized than the unspecialized, we may have identified a domain-specific term. techniques that use this statistical strategy work well. while many variables with ate methods have been carefully evaluated and tuned in the literature, the effects that come from choosing a particular background corpus over another are not obvious. more specifically, what would happen if one were to use a more broad background corpus that contained a wider variety of subject matter? this paper presents an experiment carried out with termolator @xcite , a highperforming open-source ate system. the system allows for the specification of a foreground corpus consisting of the target topic area and a background corpus that can be customized. we explore the results from running this test on three distinct patent topic areas, using the cooperative patent classification (cpc) scheme to curate five background corpora for each foreground. our results show that the choice of background corpus has a significant effect on the precision of the words extracted. for every foreground corpus, the broadest background corpus gave the worst performance, in the worst case that difference is 17%. similarly, the least broad background corpus gave suboptimal performance in all three experiments. indeed, the ideal background corpus seemed to occupy some middle position -broader than the foreground cor-pus, but not too general either. for example, we found that highest results (72% precision) for a foreground of semiconductor (h01l 21) patents was derived from a background of patents related to electricity (h), which is more general than "electric solid state devices" (hoil) and more specific than patents in general or than a combination of patents and non-patents. we perform a qualitative analysis of words extracted to see how different background breadths affect the words extracted. for example, when the top 100 term candidates from the data input patent foreground corpus were analyzed, the most general background corpus produced a set of terminology that, while technical, was less characteristic of data inputs than the all patent background corpus (e.g., the most general corpus: fingerprint sensor, social media vs. a patent corpus: focal vergence, selectable interaction element). we expect that our results will generalize to terminology extraction for other legal and technical documents and, generally, to the foreground/background approach to ate. ## related work the definition of 'terminology' in the context of ate systems is still a point of discussion in modern literature @xcite . in this study we use the word terminology to describe specialized language that is domain specific. notionally, we distinguish a word or phrase as terminology if it is sufficiently specialized that a typical naive adult would not be expected to know the meaning of the term @xcite . ate methods are generally split into 3 different categories: linguistic, statistical, and hybrid. linguistic methods use linguistic features such as parts of speech patterns and chunking to extract term candidates. statistical methods usually use a statistical measure of how characteristic a term is to a foreground corpus by comparing it to a baseline background corpus. hybrid methods combine the linguistic and statistical methods, usually by using linguistic methods to identify term candidates and the statistical methods to rank the candidates. the statistics in the hybrid methods work by comparing a foreground corpus from which terminology should be extracted, with a background corpus which serves as a baseline to identify terms characteristic of the foreground. the use of a foreground and a background corpus (or sometimes an analysis and reference corpus, respectively) has existed for a long time (e.g. @xcite @xcite ). the intuition is that by using and combining these statistics, one can rank the words and phrases which are most likely to be specialized language from the foreground higher. a variety of statistics have been used in the literature (e.g. tf-idf, kl divergence, etc.) @xcite . the assumption behind using a foreground and background corpus is that the foreground is sufficiently specialized and the background corpus is sufficiently general that the way they use potential terms will be different. this assumption is powerful and effective and has led some research to stick to a single general background corpus @xcite and some research to allow varying background corpora @xcite . by taking advantage of both linguistic and statistical techniques, hybrid methods have proven to be some of the most effective in ate for the last decade @xcite . while most systems now fall into the hybrid category, there is growing interest in machine learning methods for ate with a variety of methodologies @xcite . in this paper we use an open-source hybrid method called termolator that combines chunking and statistical ranking of term candidates using two corpora: the foreground corpus and the background corpus @xcite . termolator is a flexible hybrid ate system that allows us to vary the background corpus for a given foreground corpus. we are assuming that termolator is representative of other hybrid systems which use a foreground and background corpus in the same way. we believe this is a valid assumption because such ate systems are based on the idea that comparing the distribution of terms candidates across two different corpora helps identify them. terms that appear frequently in foreground documents but not background documents are more likely to be terms and vice versa. we do not make any assumptions about the relative performance of termolator and other comparable systems. in this work, we focus on patents, a technical document in the legal domain, and the relationship between foreground and background corpora. we examine how the choice of background corpora might affect the performance of existing systems and the output of those systems. @xcite discussed how the distance between foreground and background corpora affects terms in unspecialized corpora. however, their paper focuses on design choices to optimize ate for unspecialized corpora, like news articles. ## future work in our experiment, we used a general corpus that was composed of a mixture of oanc and a subset of general patents. we made this choice because our focus was making broader corpora not contrasting corpora. nonetheless, the effect of using a truly general corpus would be an important baseline to compare in future research. we limited our evaluation in this paper to precision and a qualitative analysis of the words themselves. we believe it would be relevant to devise a methodology that would allow us to further investigate the differences in the words extracted using the different background corpora. a relevant extension would be to perform similar experiments using other document types. for instance, a natural extension would be to perform a similar set of experiments on medical scholarly text from pubmed or wikipedia articles and examine if the trends we observed with patents remain true for other kinds of technical documents. ## conclusion in this paper we investigated how varying the breadth of the background corpus affects hybrid ate systems. after creating five background corpora for each foreground corpora using the cpc hierarchy, we ran three experiments on three different patent groups. we examined both the precision scores and the output words themselves. in this analysis, we were unable to find a single "best choice" for all patent classes. we found that for all three patent groups neither the narrowest nor the most broad background corpus achieved the best precision; rather, it was always a background corpus that consisted of patents that performed best. in addition, we found that the words we extracted varied with the background corpus we chose. for one patent class there was a clear separation between less specialized terms for the general corpus and the more specialized terms from the all patent corpus. this separation was not clear for the other two patent classes. we showed that the choice of background corpus has a significant effect on the precision of the output of an ate system. we found that optimizing for precision in all three cases meant choosing a patent only corpus. we also studied the words we extracted by comparing differences across runs. we found that the breadth of the corpora had a significant effect on the words extracted. moreover, we informally analyzed how the words from the general background corpus differed from the patent background corpus, concluding that the term candidates were on average less specialized with the general corpus.