Leveraging knowledge graphs to update scientific word embeddings using latent semantic imputation (2022)

Abstract: The most interesting words in scientific texts will often be novel or rare. This presents a challenge for scientific word embedding models to determine quality embedding vectors for useful terms that are infrequent or newly emerging. We demonstrate how Latent Semantic Imputation (LSI) can address this problem by imputing embeddings for domain-specific words from up-to-date knowledge graphs while otherwise preserving the original word embedding model. We use the MeSH knowledge graph to impute embedding vectors for biomedical terminology without retraining and evaluate the resulting embedding model on a domain-specific word-pair similarity task. We show that LSI can produce reliable embedding vectors for rare and out-of-vocabulary terms in the biomedical domain.

## introduction word embeddings are powerful representations of the semantic and syntactic properties of words that facilitate high performance in natural language processing (nlp) tasks. because these models completely rely on a training corpus, they can struggle to reliably represent words which are infrequent, or missing entirely, in that corpus. the latter will happen for any new terminology emerging after training is complete. rapid emergence of new terminology and a long tail of highly significant but rare words are characteristic of technical domains, but these terms are often of particular importance to nlp tasks within these domains. this drives a need for methods to generate reliable embeddings of rare and novel words. at the same time, there are efforts in many scientific fields to construct large, highly specific and continuously updated knowledge graphs that capture information about these exact terms. can we leverage these knowledge graphs to mitigate the short-comings of word embeddings on rare, novel and domain-specific words? we investigate one method for achieving this information transfer, latent semantic imputation (lsi) @xcite . in lsi the embedding vector for a given word, w, is imputed as a weighted average of existing embedding vectors, where the weights are inferred from the local neighborhood structure of a corresponding embedding vector, w d , in a domain-specific embedding space. we study how to apply lsi in the context of the biomedical domain using the medical subject headings (mesh) knowledge graph @xcite , but expect the methodology to be applicable to other scientific domains. word embeddings for the biomedical domain. much research has focused on how to best generate biomedical-specific embeddings and provide models to improve performance on downstream nlp tasks @xcite @xcite . work in the biomedical domain has investigated optimal hyperparameters for embedding training @xcite , the influence of the training corpus @xcite @xcite , and the advantage of subword-based embeddings @xcite . word embeddings for clinical applications have been proposed @xcite and an overview was provided in kalyan and sangeetha (2020). more recently, transformer models have been successfully adapted to the biomedical domain yielding contextual, domain-specific embedding models @xcite @xcite . whilst these works highlight the benefits of domain-specific training corpora this class of approaches requires retraining to address the oov problem. improving word embeddings using domain information. our problem task requires improving a provided embedding model for a given domain, without detrimental effects on other domains. @xcite use random walks over the mesh headings knowledge graph to generate additional training text to be used during the word embedding training. similar ideas have led to using regularization terms that leverage an existing embedding during training of a new embedding to preserve information from an original embedding during training on a new corpus @xcite . of course, these methods require the complete training of one or more embedding models. @xcite achieve a similar result more efficiently by defining a convex objective function that balances preserving an existing embedding with decreasing the distance between related vectors, based on external data sources such as a lexicon. this technique has been applied in the biomedical domain @xcite , but has limited ability to infer new vocabulary because without the contribution from the original embedding this reduces to an average of related vectors. another approach is to extend the embedding dimension to create space for encoding new information. this can be as simple as vector concatenation from another embedding @xcite , possibly followed by dimensionality reduction @xcite . alternatively, new dimensions can be derived from existing vectors based on external information like synonym pairs @xcite . again, this has limited ability to infer new vocabulary. all of these methods change the original embedding, which limits applicability in use-cases where the original embedding quality must be retained or where incremental updates from many domains are required. the optimal alignment of two partially overlapping word embedding spaces has been studied in the literature on multilingual word embeddings @xcite @xcite and provides a mechanism to patch an existing embedding with information from a domain-specific embedding. unfortunately, it assumes the embedding spaces have the same structure, meaning it is not suitable when the two embeddings encode different types of information, such as semantic information from text and relational information from a knowledge base. ## latent semantic imputation lsi, the approach pursued in this paper, represents embedding vectors for new words as weighted averages over existing word embedding vectors with the weights derived from a domain-specific feature matrix @xcite . this process draws insights from locally linear embedding @xcite . specifically, a local neighborhood in a high-dimensional word embedding space e s (s for semantic) can be approximated by a lower-dimensional manifold embedded in that space. hence, an embedding vector w s for a word w in that local neighborhood can be approximated as a weighted average over a small number of neighboring vectors. this would be useful to construct a vector of a new word w if we could determine the weights for the average over neighboring terms. but since, by assumption, we do not know w's word embedding vector w s , we also do not know its neighborhood in e s . the main insight of lsi is that we can use the local neighborhood of w's embedding w d in a domain-specific space, e d , as a proxy for that neighborhood in the semantic space of our wordembedding model, e s . the weights used for constructing an embedding for w in e s are calculated from the domain space as shown in fig. 1 : a knearest-neighbors minimum-spanning-tree (knn-mst) is built from the domain space features. then the l2-distance between w d and a weighted average over its neighbors in the knn-mst is minimized using non-negative least squares. the resulting weights are used to impute the missing embedding vectors in e s using the power iteration method. this procedure crucially relies on the existence of words with good representations in both e s and e d , referred to as anchor terms, which serve as data from which the positions of the derived embedding vectors are constructed. figure 1 : latent semantic imputation. r d is the domain space and r s is the semantic space. ## methodology we extend the original lsi procedure described above in a few key ways. instead of using a numeric data matrix as the domain data source of lsi, we use a node embedding model trained on a domain-specific knowledge graph to obtain e d . as knowledge graphs are used as a source of structured information in many fields, we expect our method to be applicable to many scientific domains. knowledge graphs are prevalent in scientific fields as they serve as a means to organise and store scientific data, as well as to aid downstream tasks such as reasoning and exploration. their structure and ability to represent different relationship types makes it relatively easy to integrate new data, meaning they can evolve to reflect changes in a field and as new data becomes available. we use the 2021 rdf dump of the mesh knowledge graph (available at https://id. nlm.nih.gov/mesh/ ). the complete graph consists of 2,327,188 nodes and 4,272,681 edges, which we reduce into a simpler, smaller, and undirected graph to be fed into a node embedding algorithm. we extract a subgraph consisting of solely the nodes of type "ns0__topicaldescriptor" and the nodes of type "ns0__concept" that are directly connected to the topical descriptors via any relationship type. the relationship types and directionality were removed. this results in 58,695 nodes and 113,094 edges. we use the node2vec graph embedding algorithm @xcite on this subgraph to produce an embedding matrix of 58,695 vectors with dimension 200 (orange squares in fig. 2 ). the hyperparameters are given in appendix 8.1. these node embeddings form the domain-specific space, e d , as described in the previous section. we note that in preliminary experiments, the adjacency matrix of the knowledge graph was used directly as e d but this yielded imputed embeddings that performed poorly. to provide the mapping between the mesh nodes and the word embedding vocabulary we normalize the human-readable "rdfs__label" node property by replacing spaces with hyphens and lower-casing. the anchor terms are then identified as the normalized words that match between the graph labels and the vocabulary of the word-embedding model; resulting in 12,676 anchor terms. as an example, "alpha-2-hs-glycoprotein" appears as both a node in the reduced graph and in the word-embedding model, along with its neighbors in the knn-mst, which include "neoglycoproteins" and "alpha-2-antiplasmin". these serve to stabilise the positions of unknown word embedding vectors for domain space nodes which did not have corresponding representations in the semantic space during the lsi procedure. lsi has one key hyper-parameter: the minimal degree of the knn-mst graph, k. the stopping criterion of the power iteration method is controlled by another parameter, η, but any sufficiently small value should allow adequate convergence and have minimal impact on the resulting vectors. following @xcite we set η = 10 -4 but we use a larger @xmath0. ## experiments we aim to answer two questions to evaluate our imputation approach: do the imputed embeddings encode semantic similarity and relatedness information as judged by domain experts? and, can the imputed embeddings be reliably used alongside the original, non-imputed word embeddings? we use the umnsrs dataset to answer these questions @xcite . it is a collection of medical word-pairs annotated with a relatedness and similarity score by healthcare professionals, such as medical coders and clinicians; some examples are shown in to obtain additional insight into the performance of the imputation procedure we split the words in the umnsrs dataset into two groups of roughly the same size: one group of words (trained) which we train directly as part of the word embedding training and another group of words (imputed) which we obtain via imputation. this split results in three word-pair subsets that contain imputed/imputed word pairs, trained/trained word pairs, and imputed/trained word pairs. note that due to an incomplete overlap of the umnsrs test vocabulary with both the mesh node labels and our word embedding vocabulary we cannot evaluate on every word pair in umnsrs (see for word embedding training we prepare a corpus of 74.4m sentences from open access publications on pubmed (from https://ftp. ncbi.nlm.nih.gov/pub/pmc/oa_bulk/ ; accessed on 2021-08-30). to simulate the problem of missing words as realistically as possible we then prepare a filtered version of this corpus by removing any sentence containing one of the imputed terms (in either singular or plural form). this filtering removes 2.36m of the 74.4m sentences (3.2%). we then train 200-dimensional skip-gram word embedding models on both the full and the filtered version of the training corpus. in addition, we also train fasttext embeddings @xcite on both the full and the filtered corpus. for details on the hyper-parameters see appendix 8.2. since fasttext, which represents words as n-grams of their constituent characters, has been shown to give reasonable embedding vectors for words which are rare or missing in the training corpus it represents a suitable baseline to which we can compare our imputation procedure. we check that the embedding models (both skip-gram and fasttext) trained on the filtered corpus perform roughly on par with those trained on the full corpus when evaluated using the trained/trained subset of the umnsrs test data. we also check that the skip-gram model trained on the full corpus performs comparable to the biowordvec model @xcite across all subsets of umnsrs. see appendix 8.3 for details. lsi is a means of leveraging the domain space to create oov embedding vectors. as a simple alternative baseline, we directly use the domain space embeddings for the oov words. we need to align the domain space onto the semantic space, which we do with a rotation matrix derived from the anchor term embeddings in the two spaces via singular value decomposition. ## conclusion in this paper, we show how lsi can be used to improve word embedding models for the biomedical domain using domain-specific knowledge graphs. we use an intrinsic evaluation task to demonstrate that lsi can yield good embeddings for domainspecific out of vocabulary words. we significantly extend the work of @xcite by showing that lsi is applicable to scientific text where problems with rare and novel words are particularly acute. @xcite assumed a small number of domain entities and a numeric domain data feature matrix. this immediately yields the metric structure required to determine the nearest neighbors and minimum spanning tree graph used in lsi. we extend this to a much larger number of domain entities and to domain data which does not have an a priori metric structure but is instead given by a graph structure. we demonstrate that lsi can also work with relational domain data thus opening up a broader range of data sources. the metric structure induced by node embeddings trained on a domain knowledge graph provides an equally good starting point for lsi. this shows that lsi is a suitable methodology for controlled updates and improvements of scientific word embedding models based on domainspecific knowledge graphs.