Unsupervised Term Extraction for Highly Technical Domains (2022)

Abstract: Term extraction is an information extraction task at the root of knowledge discovery platforms. Developing term extractors that are able to generalize across very diverse and potentially highly technical domains is challenging, as annotations for domains requiring in-depth expertise are scarce and expensive to obtain. In this paper, we describe the term extraction subsystem of a commercial knowledge discovery platform that targets highly technical fields such as pharma, medical, and material science. To be able to generalize across domains, we introduce a fully unsupervised annotator (UA). It extracts terms by combining novel morphological signals from sub-word tokenization with term-to-topic and intra-term similarity metrics, computed using general-domain pre-trained sentence-encoders. The annotator is used to implement a weakly-supervised setup, where transformer-models are fine-tuned (or pre-trained) over the training data generated by running the UA over large unlabeled corpora. Our experiments demonstrate that our setup can improve the predictive performance while decreasing the inference latency on both CPUs and GPUs. Our annotators provide a very competitive baseline for all the cases where annotations are not available.

## introduction automated term extraction (ate) is the task of extracting terminology from domain-specific corpora. term extraction is the most important information extraction task for knowledge discovery systems -whose aim is to create structured knowledge from unstructured text -because domain specific terms are the linguistic representation of domainspecific concepts. to be of use in knowledge discovery systems (e.g., saga @xcite , deepsearch @xcite ) the term extraction has to identify individual mentions of terms to enable downstream components (i.e., the entity jpeg (/ˈdʒeɪpɛɡ/ jay-peg) [2] is a commonly used method of lossy compression for digital images, particularly for those images produced by digital photography. ## related work automated term extraction (ate) is a natural language processing task that has been the subject of many research studies @xcite @xcite @xcite . what we describe in this work is an effective term extraction approach that is fully unsupervised and also offers the flexibility and modularity to deploy and easily maintain systems in production. ate should not be confused with keyphrase extraction @xcite @xcite and keyphrase generation @xcite , which have the goal of extracting, or generating, key phrases that best describe a given free text document. keyphrases can be seen as a set of tags associated to a document. in the context of keyphrase extraction, sentence embedders have been used in the literature, such as in embedrank @xcite and key2vec @xcite . in our work, we also rely on sentence encoders, but we use them to generate training data for sequence tagging. therefore, we do not rely on sentence encoders at runtime to extract terminology from text, enabling the creation of lower latency systems. to capture complex morphological structures we use word segmentation techniques. word segmentation algorithms such as byte-pair encoding @xcite , word-piece @xcite , and unigram language modeling @xcite have been introduced to avoid the problem of out-of-vocabulary words and, more in general, to reduce the number of distinct symbols that sequence models for natural language processing have to process. to the best of our knowledge, we are the first to use the subword-unit tokenization as a signal to extract technical terms from text. our approach builds on the notion of specificity to find terminology. while there are multiple research works @xcite highlighting the importance of specificity, to the best of our knowledge, this is the first work using the notion of specificity to extract terminology from text. raw corpus (no labels) unsupervised annotator (ua) generated annotated data transformer fine-tuning final model for inference weak supervision unsupervised figure 2: our training workflow consists of 1) generating training data from raw unlabeled text using our unsupervised annotator, and 2) fine-tuning a transformerbased model or any sequence tagging model. ## the approach annotator (ua), which, given the raw corpus, produces a training dataset for sequence labeling. the resulting dataset is used to train (or fine-tune) a sequence model that represents the final model for term annotation used at inference time. pre-trained transformer-based models clearly represent a valid alternative to implement such sequence models. moreover, we can avoid pre-training since the ua potentially generates a large amount of training data. from the software engineering standpoint, this setup is extremely attractive as it makes the architecture of the term extraction subsystem modular and very flexible. the modularity comes from decoupling the inference component and the unsupervised annotator (ua). the unsupervised annotator can be enhanced with additional and more computationally demanding subcomponents (e.g., an entity linker to an external knowledge base), without increasing the final inference latency observed by the user. this modularity enables domain customization with proprietary data (and systems), which might be available for specific domains or customers. since the integration between the unsupervised annotator and the inferencing component is achieved via data (i.e., the training samples for sequence tagging expressed in iob format) the approach enables the smooth transition between a fully unsupervised setup and a setup where manual annotations augment the ones obtained via the ua. in practice, in realistic deployments, the unsupervised annotator is used to boostrap the term extraction subsystem, while domain specific annotations are added over time by organizing annotation campaigns or by collecting labels through the interactions of the users with the knowledge discovery platform. having a dedicated component for inferencing, which is independent from the ua, gives the flexibility to select the right trade-off in terms of accuracy, inference latency, deployment costs, and inferencing infrastructure. this choice is completely independent from the unsupervised annotator, which can be independently improved without taking care of inference latency. since the inference component can be built around off-theshelf transformer-based models, one can fully leverage the optimizations available in modern commercial offerings for inferencing services (e.g., amazon sagemaker, huggingface infinity). as transformer-based models are frequently used for multiple tasks (e.g., classification, ner, qa) within a knowledge discovery platform, this often corresponds to having a very homogeneous inferencing infrastructure in production. however, given that the ua can potentially generate a large amount of training samples, large pre-trained models are not a necessity, and even alternative architectures such as pqrnn @xcite or pnlp-mixer @xcite ) can be used. ## experiments we now assess whether our approach can represent a valid baseline for term extraction in different technical domains when annotated data is not available. we aim to answer the following research questions: ## conclusion in this paper, we described an effective term extraction approach that uses a fully unsupervised annotator to generate training data to fine-tune transformer models. this approach reduces the inference time of the unsupervised annotator, without decreasing its performance, and allows the flexibility to pick the right trade-off between latency and f1 score. the latency-optimized models are less than 30 megabytes in size, provide inference latencies lower then 30 ms even without gpus, while exhibiting a competitive f1 score compared to the models fine-tuned with manually annotated data.