Extracting Text Representations for Terms and Phrases in Technical Domains (2023)

Abstract: Extracting dense representations for terms and phrases is a task of great importance for knowledge discovery platforms targeting highly-technical fields. Dense representations are used as features for downstream components and have multiple applications ranging from ranking results in search to summarization. Common approaches to create dense representations include training domain-specific embeddings with self-supervised setups or using sentence encoder models trained over similarity tasks. In contrast to static embeddings, sentence encoders do not suffer from the out-of-vocabulary (OOV) problem, but impose significant computational costs. In this paper, we propose a fully unsupervised approach to text encoding that consists of training small character-based models with the objective of reconstructing large pre-trained embedding matrices. Models trained with this approach can not only match the quality of sentence encoders in technical domains, but are 5 times smaller and up to 10 times faster, even on high-end GPUs.

## introduction large pre-trained language models are extensively used in modern nlp systems. while the most typical application of language models is fine-tuning to specific downstream tasks, language models are often used as text encoders to create dense features consumed by downstream components. among the many use cases of dense text representations there is search, question answering, and classification @xcite . static embeddings, trained with algorithms such as word2vec @xcite , can exploit existing information extraction pipelines to create representations for entities, phrases, and terms present in text corpora. static embedding matrices are trained with self-supervised approaches at regular intervals, either when additional data is avail-* equal contribution. in contrast, contextualized word embeddings leverage sentence encoders @xcite to dynamically create dense representations for any input text by performing a forward pass over a large language model. specifically, a word embedding is computed at inference time based on its context, unlike static word embeddings that have a fixed (contextindependent) representation. in practice, sentence encoders solve the out-of-vocabulary (oov) problem which affects static embeddings at the cost of high computational requirements and stronger dependencies on supervised datasets for similarity. despite the popularity of sentence encoders, large pre-trained embedding matrices are still widely adopted in the industry to encode not only individual tokens but also multi-token entities extracted with in-house nlp pipelines. once those embedding matrices are trained, the text representation for single-and multi-token entries encountered at training time can be looked up in constant time. in this paper, we describe an effective approach taken to provide high-quality textual representations for terms and phrases in a commercially available platform targeting highly-technical domains. our contribution is a novel unsupervised approach to train text encoders that bridges the gap between large pre-trained embedding matrices and computationally expensive sentence encoders. in a nutshell, we exploit the vast knowledge encoded in large embedding matrices to train small character-based models with the objective of reconstructing them, i.e., we use large embedding matrices trained with self-supervision as large training datasets mapping text to the embedding vectors. our approach is extremely attractive for industrial setups as it leverages continuous improvements, testing, and inference costs of existing information extraction pipelines to create large datasets to train text encoders. this way, the return on investment for annotations, development, and infrastructure costs are maximized. in our evaluation, we highlight that by combining unsupervised term extraction annotators and static embeddings we can train lightweight character-based models that match the quality of supervised sentence encoders and provide substantially better representations than sentence encoders trained without supervision. our models not only provide competitive representations, but are up to 5 times smaller and 10 times faster than sentence encoders based on large language models. ## existing approaches the mission of our industrial knowledge discovery platform is to extract knowledge from large corpora containing highly-technical documents, such as patents and papers, from diverse fields ranging from chemistry, to physics, to computer science. information extraction is extremely challenging given the large variety of language nuances and the large cost of human annotations in such specialized domains. therefore, it is of extreme importance to minimize the dependencies on annotated data and to use unsupervised approaches whenever possible. a recurring requirement from many internal components of our platform is the ability to extract high-quality dense representations for technical terms, entities, or phrases which can be encountered in many distinct technical fields. high-quality representations are extremely valuable to implement semantic search, to influence the ranking, or to be used directly as model features. in modern industrial systems, it is often the case that static and context-dependent embedding technologies coexist on the same platform to extract representations. while static embeddings are trained in a self-supervised fashion, sentence encoders are often built by fine-tuning pre-trained models on similarity tasks using annotated datasets. having two separate approaches for text encoding is suboptimal as those systems are completely independent and embed terms into distinct embedding spaces. to reconcile those two worlds, we propose an approach where static embeddings and text en-coders are mapping text into the same embedding space. our intuition is that static embedding matrices storing embeddings for single tokens, but also multi-token terms, entities, or phrases, represent an invaluable source of information to train text encoders. while those matrices are built with selfsupervision, they can leverage existing annotators, supervised or not, which are commonly available in large text processing platforms. our novel approach consists of using pre-trained embedding matrices as a training set, and training character-based models, called charemb, to predict the embedding vector for a text. this means that the character-based models will enable to project any sequence of characters in the same embedding space as the pre-trained embedding matrices. ## our model: charemb instead of having two completely disjoint systems to create text representations, we use characterbased models trained over large static embedding matrices, that project a sequence of text into the same embedding space as the embedding matrices used as training data. in practice, we approach text encoding as a compression problem, where character-based models are trained to reconstruct the pre-trained static embedding matrices, as shown in figure 1 . this training approach can rely on supervised sequence tagging models or can be implemented using fully unsupervised methods, such as the term extraction technologies described in the literature @xcite . as we highlight in the evaluation section, a text encoder trained without any supervision can match the performance of supervised sentence encoders in creating representations for words or phrases in technical domains. to train our models, we consider a static pretrained embedding matrix as gold dataset. an individual training sample associates a single-or multitoken text to an embedding vector. to leverage the dataset we train a text encoder model to minimize the cosine similarity between the produced vector and the original vector stored in the static embedding matrix. the models rely on character-level tokenization to generalize better on unseen inputs in technical domains. in characters, for which an embedding matrix is trained. character-level embeddings are used as input for a long short-term memory sequence model. the last state of the lstm layer is used to produce, via a multi-layer perceptron, a vector of dimension k that represents the embedding for the text t. the network is trained to minimize the cosine distance between the predicted embedding and the original one stored in the embedding matrix. the number of distinct training samples is |v |, which means that embeddings with large vocabularies correspond to bigger training datasets. ## evaluation in this section, we compare our text representations using the patent phrase similarity dataset built by google @xcite . given two multiword technical concepts (called anchor and target), the task consists of predicting a similarity score between the two terms. unlike generalpurpose sentence datasets, such as sts-b @xcite or sick @xcite , we focus on technical concepts in the patent and scientific domains. the dataset contains 38, 771 unique concepts and 36, 473, 2, 843, and 9, 232 concept pairs with humanly-annotated similarity scores for the train, validation, and test sets, respectively. in our case, we are only interested in the zeroshot scenario, and thus, we only consider the test set and ignore the train and validation sets. we evaluate the quality of the text representations using the same approach described in aslanyan and wetherbee (2022): we compute the pearson and spearman correlation between the cosine similarity of the two embeddings and the human-annotated scores. foot_0 ## related work static embeddings trained with self-supervised setups became popular with word2vec @xcite and glove @xcite . while those algorithms have been originally introduced to embed individual tokens, the approach can be generalized to entire phrases or multiple token entities by preprocessing training corpora such that multiple tokens are merged into one. fasttext @xcite can be seen as an extension to word2vec which relies on n-grams to extract representations of unseen text. contextualized embeddings (e.g., elmo @xcite ) are created by taking into account the contex of each token. sentence encoders @xcite are a generalization of contextual embeddings. they can be trained on sentence-level tasks using supervised datasets, such as nli, or with unsupervised methods @xcite . our method to train text encoders is fully unsupervised and provides higherquality representations than supervised encoders. embedding compression is a topic of great interest not only for natural language processing @xcite , but also in recommender systems @xcite @xcite . the primary goal of our work is not to reduce the size of a static embedding matrix, but rather to generalize the embeddings to entries not seen at training time. work has been done to align embedding spaces coming from different models @xcite @xcite . instead of aligning spaces coming from static embeddings and sentence encoders, we introduce text encoders trained to project text in the same space of a static embedding matrix used as a training dataset. ## conclusion creating embeddings for terms and phrases is of paramount importance for complex natural language processing platforms targeting highlytechnical domains. while out-of-the-box pretrained sentence encoders are often considered as baselines, representations of similar quality can be obtained with substantially lighter and simpler character-based models which are 5 times smaller in size and 10 times faster at inference time, even on high-end gpus. the key to obtaining such results is to realize that large static embedding matrices storing representations for tokens and terms constitute a very rich supervised dataset to train text encoders working at the character level. since both term extraction and embedding training can be per-formed without any labeled data, we have proposed a method to train text encoders which does not require any label. those models are trained with the objective of reconstructing the original embedding matrix and can not only be used as lighter alternatives to sentence encoders, but also as lossless compressors for large embedding matrices. of much larger embedding matrices. third, our charemb model achieves the best performance when trained with an embedding matrix containing embeddings for terms. predicting the embeddings with our charemb model allows to achieve significantly higher correlation than the original matrix containing terms.