Improving Biomedical Analogical Retrieval with Embedding of Structural Dependencies (2020)

Abstract: Inferring the nature of the relationships between biomedical entities from text is an important problem due to the difficulty of maintaining human-curated knowledge bases in rapidly evolving fields. Neural word embeddings have earned attention for an apparent ability to encode relational information. However, word embedding models that disregard syntax during training are limited in their ability to encode the structural relationships fundamental to cognitive theories of analogy. In this paper, we demonstrate the utility of encoding dependency structure in word embeddings in a model we call Embedding of Structural Dependencies (ESD) as a way to represent biomedical relationships in two analogical retrieval tasks: a relationship retrieval (RR) task, and a literature-based discovery (LBD) task meant to hypothesize plausible relationships between pairs of entities unseen in training. We compare our model to skip-gram with negative sampling (SGNS), using 19 databases of biomedical relationships as our evaluation data, with improvements in performance on 17 (LBD) and 18 (RR) of these sets. These results suggest embeddings encoding dependency path information are of value for biomedical analogy retrieval.

## introduction distributed vector space models of language have been shown to be useful as representations of relatedness and can be applied to information retrieval and knowledge base augmentation, including within the biomedical domain @xcite . a vast amount of knowledge on biomedical relationships of interest, such as therapeutic relationships, drug-drug interactions, and adverse drug events, exists in largely human-curated knowledge bases @xcite . however, the rate at which new papers are published means new relationships are being discovered faster than human curators can manually update the knowledge bases. furthermore, it is appealing to automatically generate hypotheses about novel relationships given the information in scientific literature @xcite , a process also known as 'literaturebased discovery.' a trustworthy model should also be able to reliably represent known relationships that are validated by existing literature. neural word embedding techniques such as word2vec foot_0 and fasttext foot_1 are a widely-used and effective approach to the generation of vector representations of words @xcite and biomedical concepts @xcite ). an appealing feature of these models is their capacity to solve proportional analogy problems using simple geometric operators over vectors @xcite . in this way, it is possible to find analogical relationships between words and concepts without the need to specify the relationship type explicitly, a capacity that has recently been used to identify therapeutically-important drug/gene relationships for precision oncology @xcite . however, neural embeddings are trained to predict co-occurrence events without consideration of syntax, limiting their ability to encode information about relational structure, which is an essential component of cognitive theories of analogical reasoning @xcite . additionally, recent work @xcite has found that contextualized word embeddings from language models such as elmo, when evaluated on analogy tasks, perform worse on semantic relation tasks than static embedding models. the present work explores the utility of encoding syntactic structure in the form of dependency paths into neural word embeddings for analogical retrieval of biomedical relations. to this end, we build and evaluate vector space models for representing biomedical relationships, using a corpus of dependency-parsed sentences from biomedical literature as a source of grammatical representations of relationships between concepts. we compare two methods for learning biomedical concept embeddings, the skip-gram with negative sampling (sgns) algorithm @xcite and embedding of semantic predications (esp) @xcite , which adapts sgns to encode concept-predicate-concept triples. in the current work, we adapt esp to encode dependency paths, an approach we call embedding of structural dependencies (esd). we train esd and sgns on a corpus of approximately 70 million sentences from biomedical research paper abstracts from medline, and evaluate each model's ability to solve analogical retrieval problems derived from various biomedical knowledge bases. we train esd on concept-path-concept triples extracted from these sentences, and sgns on full sentences that have been minimally preprocessed with named entities (see §3). figure 1 shows the pipeline from training to evaluation. from an applications perspective, we aim to evaluate the utility of these representations of relationships for two tasks. the first involves correctly identifying a concept that is related in a particular way to another concept, when this relationship has already been described explicitly in the biomedical literature. this task is related to the nlp task of relationship extraction, but rather than considering one sentence at a time, distributional models represent information from across all of the instances in which this pair have co-occurred, as well as information about relationships between similar concepts. we refer to this task as relationship retrieval (rr). the second task involves identifying concepts that are related in a particular way to one another, where this relationship has not been described in the literature previously. we refer to this task as literature-based discovery (lbd), as identifying such implicit knowledge is the main goal of this field @xcite . we evaluate on four kinds of biomedical relationships, characterized by the semantic types of the entity pairs involved, namely chemical-gene, chemical-disease, gene-gene, and gene-disease relationships. the following paper is structured as follows. §2 describes vector space models of language as they are evaluated for their ability to solve proportional analogy problems, as well as prior work in encoding dependency paths for downstream applications in relation extraction. §3 presents the dependency path corpus from @xcite . §4 summarizes the knowledge bases from which we develop our evaluation data sets. §5 describes the training details for each vector space model. §6 and §7 describe the methods and results for the rr and lbd evaluation paradigms. §8 and §9 offer discussion and conclude the paper. code and evaluation data will be made available at https://github.com/amandalynne/esd . ## background we look to prior work in using proportional analogies as a test of relationship representation in the general domain with existing studies on vector space models trained on generic english. while our biomedical data is largely in english, we constrain our evaluation to specific biomedical concepts and relationships as we apply and extend established methods. ## text data we train both the esd and sgns models on data released by @xcite . this corpus foot_3 consists of about 70 million sentences from a subset of medline (approximately 16.5 million abstracts) which have pubtator @xcite annotations applied to identify phrases that denote names of chemicals (including drugs and other chemicals of interest), genes (and the proteins they code for), and diseases (including side effects and other phenotypes). throughout this paper, we use these shorthand names for each of these categories, following the convention established in @xcite and followed by percha and altman (2018). the following example sentence from an article processed by pubtator shows how multi-word phrases that denote biomedical entities of interest, in this case atypical depression and seasonal affective disorder, are concatenated by underscores to constitute single tokens: chromium has a beneficial effect on eating-related atypical symptoms of depression, and may be a valuable agent in treating atypical depression and seasonal affective disorder. percha and altman (2018) also provide pruned stanford dependency @xcite parses for the sentences in the corpus, consisting, for each sentence, of the minimal path of dependency relations connecting pairs of biomedical named entities identified by pubtator. specifically, they extract dependency paths that connect chemicals to genes, chemicals to diseases, genes to diseases, and genes to genes. figure 2 shows an example of a dependency path of relations between two terms, risperidone and rage. we use these dependency paths as representations for predicates that denote biomedical relationships of interest by concatenating the string representations of each path element, which are shown below the sentence in figure 2 . following percha and altman (2018), we exclude paths that denote a coordinating conjunction between elements and paths that denote an appositive construction, both of which are highly common in the set. in this corpus of 70 million sentences, there are about 44 million unique dependency paths that connect concepts of interest, the vast majority (around 40 million) of which appear just once in the corpus. 540,011 of these paths appear at least 5 times in the corpus. ## knowledge bases we construct our evaluation data sets with exemplars from knowledge bases for four primary kinds of biomedical relationships, characterized by the interactions between pairs of entities of the following types: chemical-gene, chemical-disease, gene-disease, and gene-gene. we evaluate on pairs of entities from the following knowledge bases: drugbank (wishart et al., 2018), online mendelian inheritance in man (omim) @xcite , pharmgkb (pgkb) @xcite , reactome @xcite , side effect resource (sider) @xcite , and therapeutic target database (ttd) @xcite . each knowledge base consists of pairs of entities that relate in a specific way. for example, sider side effects consists of chemical-diseasetyped pairs such that the chemical is known to have the disease as a side effect, e.g. (sertraline, insomnia). meanwhile, another chemical-disease pair from a different database, therapeutic target database (ttd) indications, is such that the chemical is indicated as a treatment for the disease, e.g. (carphenazine, schizophrenia). in constructing our evaluation sets, we process all terms such that they are lower-cased, and multi-word terms are concatenated by underscores. furthermore, we eliminate from our evaluation sets any knowledge base terms that do not appear in the training corpus described in §3 at least 5 times. it should be noted that across these sets, a single biomedical entity may appear with numerous spellings and naming conventions. esd with esd, a shallow neural network is trained to estimate the probability of encountering the object, o, of a subject-predicate-object triple sp o. the training objective involves maximizing this probability for true objects p (o|s, p ) and minimizing it for randomly drawn counterexamples, ¬o, p (¬o|s, p ). we adapted the semantic vectors 5 implementation of esp to encode dependency paths, with binary vectors as representational basis @xcite and the non-negative normalized hamming distance (n n hd) to estimate the similarity between them. @xmath0, 1 - ## × hamming distance dimensionality with this representational paradigm, probability can be estimated as n n hd(o, s ⊗ p ), where ⊗ represents the use of pairwise exclusive or as a binding operator, in accordance with the binary spatter code @xcite . while esp was originally developed to encode knowledge extracted from the literature using a small set of predefined predicates (e.g. treats), we adapt it here to encode a large variety (@xmath1,085) of dependency paths. for training, we concatenate the dependency relations (the underscored parts in figure 2 ) into a single predicate token for which a vector is learned. some examples of path tokens (concatenated dependency relations) can be seen in both sgns and esd were trained over five epochs, with a subsampling threshold of 10 -5 , a minimum term frequency threshold of 5 (which includes concatenated dependency paths for esd), and a maximum frequency threshold of 10 6 . ## evaluation methods we use a proportional analogy ranked retrieval task for both the rr and lbd tasks, following prior work as described in §2. figure 3 visualizes this process. from a set of (x, y) entity pairs from a knowledge base, given a term c and all terms d such that (c, d) is a pair in the set, we select n random (a, b) cue pairs from a disjoint set of pairs. we refer to (c, d) pairs as 'target pairs,' correct d completions as 'targets,' and (a, b) pairs as 'cues.' the vectors for the cue terms (a, b) and the term c are summed in the following fashion to produce the resulting vector v. given an analogical pair a:b::c:d, where a and c, b and d are of the same semantic type, respectively, we develop cue vectors for the target d in each model as follows: score path 0.974 controlled nmod start entity end entity amod controlled 0.935 add-on nmod start entity end entity amod add-on 0.565 reduces nsubj start entity reduces dobj requirement requirement nmod end entity 0.537 associated compound start entity end entity nsubj associated 0.516 start entity conj efficacy efficacy acl treating treating dobj end entity 0.438 treatment amod start entity treatment nmod end entity a k-nearest neighbor search is performed for v (using cosine distance for sgns, nnhd for esd) over the search space, and we record the ranks for each correct d target. the search space is constrained such that it consists of those terms from our training corpus that have a vector in both esd and sgns, a total of about 300,000 terms overall. for esd, this space consists of the output weight vectors for each concept. for the proportional analogy task using k-nearest neighbors to rank completions to the analogy, the desired outcome is for the correct targets to be highly similar to the analogy cue vector v, such that the highest ranks are assigned to the correct target terms d in a search over the entire vector space. in this fashion, we perform this knn search for every (x, y) pair in the knowledge base and record the ranks for correct targets. we then compare the ranks of terms d across both vector spaces; the higher the ranks, the better the model is at capturing relational similarity. we examine the role of increasing the number of cues in improving retrieval. for example, for a given (c, d) target pair, we can combine vectors for multiple (a, b) pairs with the c term vector to produce a final cue vector that is closer to the target d. when multiple cues are used, we superpose the cue vector for each of the cues, and normalize the resulting vector, with normalization of real vectors to unit length in sgns, and normalization of binary vectors using the majority rule with ties split at random with esd. cues are always selected from the subset of knowledge base pairs that co-occur in our training corpus. we ensure that none of the (a, b) cue terms overlap with each other, nor with the (c, d) target terms, to assure that self-similarity does not inflate performance. we produced results for a range of 1, 5, 10, 25, and 50 cues, finding that the best results come from using 25 cues; we only report these resulting scores in §7. as a baseline inspired partly by @xcite , we compute the similarity of vectors for b and d terms and c and d terms compared directly to each other, omitting the analogical task. the intuition here is that c and d terms are potentially close together in the vector space merely due to frequent co-occurrence in the corpus, and any analogical reasoning performance is merely relying on that fact. meanwhile, terms b and d can be close together in the vector space simply because they are the same semantic type, and thus occur in similar contexts. in this case, relational analogy might not explain the performance, but mere distributional similarity. in the b:d comparison setting, cues b are added together to create a single cue vector with which to perform the knn ranking over terms in which to find the target term d. these cue terms b are extracted from the same a, b cue pairs as those used for the full analogy setting to ensure a reasonable comparison across methods. in the c:d comparison setting, no cues are aggregated. ## results we present qualitative and quantitative results for each vector space model's ability to represent and retrieve relational information. ## discussion the results in ## conclusion we have compared two vector space models of language, embedding of structural dependencies and skip-gram with negative sampling, for their ability to represent biomedical relationships from literature in an analogical retrieval task. our results suggest that encoding structural information in the form of dependency paths connecting biomedical entities of interest can improve performance on two analogical retrieval tasks, relationship retrieval and literature-based discovery. in future work, we would like to compare our methods with knowledge base completion techniques using contextualized vectors from language models as in @xcite as another method applicable to literaturebased discovery.