A Systematic Investigation of KB -Text Embedding Alignment at Scale (2021)

Abstract: Knowledge bases (KBs) and text often contain complementary knowledge: KBs store structured knowledge that can support long range reasoning, while text stores more comprehensive and timely knowledge in an unstructured way. Separately embedding the individual knowledge sources into vector spaces has demonstrated tremendous successes in encoding the respective knowledge, but how to jointly embed and reason with both knowledge sources to fully leverage the complementary information is still largely an open problem. We conduct a large-scale, systematic investigation of aligning KB and text embeddings for joint reasoning. We set up a novel evaluation framework with two evaluation tasks, few-shot link prediction and analogical reasoning, and evaluate an array of KB-text embedding alignment methods. We also demonstrate how such alignment can infuse textual information into KB embeddings for more accurate link prediction on emerging entities and events, using COVID-19 as a case study.

## introduction recent years have witnessed a rapid growth of knowledge bases (kbs) such as freebase @xcite , dbpedia @xcite , yago @xcite and wikidata @xcite . these kbs store facts about real-world entities (e.g. people, places, and things) in the form of rdf triples, i.e. (subject, predicate, object). today's kbs are massive in scale. for instance, freebase contains over 45 million entities and 3 billion facts involving a large variety of relations. such large-scale multi-relational knowledge provides a great potential for improving a wide range of tasks, from information retrieval @xcite , article: united nations secretariat headquartered in new york, the secretariat functions through duty stations in addis ababa, bangkok, beirut, geneva, nairobi, santiago and vienna, in addition to offices all over the world question answering @xcite to biological data mining @xcite . kb embedding models @xcite @xcite embed entities and relations into vector space(s) such that the embeddings capture the symbolic knowledge present in the kb. similarly, word embedding models @xcite learn continuous vector representations that capture the distributional semantics of words. experiments on analogical reasoning @xcite and multilingual word embedding alignment @xcite have shown that there exists a linear structure in the word embedding space encoding relational information. on the other hand, translation-based kb embedding models @xcite @xcite , by construction, also present a linear structure in their embedding space. a natural question then is, can we align the two embedding spaces such that they mutually enhance each other? such alignment could poten-tially inject structured knowledge from kbs into text embeddings and inject unstructured but more timely-updated knowledge from text into kb embeddings, leading to more universal and comprehensive embeddings (figure 1 ). several studies have attempted at this. @xcite use the path-ranking algorithm @xcite on combined text and kb to improve binary relation prediction. @xcite leverage text data to enhance kb inference and help address the incompleteness of kbs. @xcite augment the kb with facts and relations from the text corpus and learn joint embedding for entities, kb relations and textual relations. enhancement of kb entity embeddings using using entity descriptions has been attempted in @xcite . @xcite propose to jointly embed entities and words in the same vector space. the alignment of embeddings of words and entities is accomplished using wikipedia anchors or entity names. however, existing studies are still ad-hoc and a more systematic investigation of kb-text embedding alignment is needed to answer an array of important open questions: what is the best way to align the kb and text embedding spaces? to what degree can such alignment inject information from one source to another? how to balance the alignment loss with the original embedding losses? in this work, we conduct a systematic investigation of kb-text embedding alignment at scale and seek to answer these questions. our investigation uses the latest version of the full wikidata @xcite as the kb, the full wikipedia as the text corpus, and the shared entities as anchors for alignment. we define two tasks, few-shot link prediction and analogical reasoning, to evaluate the effectiveness of injecting text information into kb embeddings and injecting kb information into text embeddings, respectively, based on which we evaluate and compare an array of embedding alignment methods. the results and discussion present new insights about this important problem. finally, using covid-19 as a case study, we also demonstrate that such alignment can effectively inject text information into kb embeddings to complete kbs on emerging entities and events. in summary, our contributions are three-fold: recently, many research works have focused on the alignment of embedding spaces of heterogeneous data sources such as different kbs. je @xcite introduces a projection matrix to align the embedding spaces of different kbs. mtranse @xcite first learns the embeddings of entities and relations in each language independently and then learns the transformation between these embedding spaces. wang et al. (2018) use graph convolutional networks and a set of prealigned entities to learn embeddings of entities in multilingual kbs in a unified vector space. in the present work, we focus on aligning the kb and textual embedding spaces. kb-text joint representation. many recent approaches have attempted to learn the embeddings of words and knowledge base entities in the same vector space. wang et al. (2014) propose an alignment technique for kb and text representations using entity names and/or anchors. wikipedia2vec (yamada et al., 2016) extends the skip-gram based model by modeling entity-entity co-occurrences using a link graph and word-entity co-occurrences using kb anchors. however, an entity mention can be ambiguous i.e. it can refer to different entities in different contexts. to resolve this, cao et al. (2017) propose multi-prototype entity mention embedding model to learn representations for different senses of entity mentions. it includes a mention sense embedding model which uses context words and a set of reference entities to predict the actual entity referred to by the mention. despite this progress, a comprehensive investigation of the merits of different alignment approaches is missing. our work takes a step forward in this direction and proposes a novel evaluation framework to compare multiple alignment approaches for kb-text joint embedding on a large-scale kb and textual corpus. ## model in this section, we describe the four alignment methods used in our study. at first, we describe the component models used in all alignment methodsthe kb embedding model and the skip-gram model. ## dataset we use wikipedia as the text corpus and wikidata @xcite as the knowledge base. we use the wikidata version dated 16 december 2020 and the wikipedia version dated 3 december 2020 for all of our experiments. the term support set (as used in the subsequent sections), denoted by s, is used to refer to the intersection set of wikidata entities and entities in wikipedia for which an article is present. dataset preprocessing. we pre-process the original set of wikidata triples and filter out entities and relations with frequency less than 10 and 5 respectively. this results in a kb with 14.64 m entities, 1222 relations, and 261 m facts. similarly, we preprocess wikipedia and filter out words from the vocabulary with frequency less than 10. however, we utilize the entire entity set of wikipedia to maximize the size of the support set. after processing, the wikipedia vocabulary consists of 2.1 m words and 12.3 m entities. ## conclusion in this work, we presented a systematic study of different alignment approaches that can be applied to align entity representations in a knowledge base and textual corpora. by evaluating on the few-shot link prediction task and analogical reasoning task, we found that although all approaches have the desired outcome, i.e., to incorporate information from the other modality, some approaches perform better than others on a particular task. we also analyzed the impact of different factors such as the size of the training set, the presence of test set entities in the support set, and the balance parameter on the evaluation task performance. we believe our evaluation framework, as well as jointly trained embeddings can serve as a useful resource for future research and applications.