On the Complementary Nature of Knowledge Graph Embedding, Fine Grain Entity Types, and Language Modeling (2020)

Abstract: We demonstrate the complementary natures of neural knowledge graph embedding, fine-grain entity type prediction, and neural language modeling. We show that a language model-inspired knowledge graph embedding approach yields both improved knowledge graph embeddings and fine-grain entity type representations. Our work also shows that jointly modeling both structured knowledge tuples and language improves both.

## introduction the surge in large knowledge graphs-e.g., freebase @xcite , dbpedia @xcite , yago @xcite -has induced knowledge graph-based applications. properly making use of this structured knowledge is a prime challenge. knowledge graph embedding @xcite @xcite addresses this problem by representing the nodes (entities) and their edges (relations) in a continuous vector space. learning these representations deduces new facts from and identifies dubious entries in the knowledge base. it also improves relation extraction @xcite , knowledge base completion @xcite and entity resolution @xcite . entity typing can provide crucial constraints and information on the knowledge contained in a kg. while historically this has been modeled as explicitly structured knowledge, and recent work has modeled the contextual language in order to make in-context entity type classifications, we argue that language modeling techniques provide an effective approach for modeling both the explicit and implicit constraints found in both structured resources and free-form contextual language. meanwhile, while language modeling [lm] has historically been a core problem within natural language processing @xcite , recent deep barack obama united states washington dc hawaii lawyer /person/politician birthplace is a type profession lives in location joint-kg-fnet-lm barack obama gave a speech to congress barack obama gave a speech to congress in washington dc sentence generation figure 1: our joint learning framework learns the representation for the entity "barack obama's" in the same embedding space as that of the given input contextual description, "barack obama gave a speech to congress." further, by learning the entity type of '/person/politician', the model provides a better contextual understanding of the underlying entity. building off of insights and advances in knowledge graph embedding, entity typing, and language modeling, we identify and advocate for leveraging the complementary nature of knowledge graphs, entity typing, and language modeling. in it, we introduce a comparatively simple framework that uses powerful, yet well-known, neural building blocks to (jointly) learn representations that simultaneously capture (1) explicit facts and information stored in a knowledge base, (2) explicit constraints on facts (exemplified by entity typing), and (3) implicit knowledge and constraints communicated via natural language and discourse. figure 1 provides an overview of the joint learning framework proposed in this work: an entity ("barack obama") along with its relations are represented in a continuous vector space. the framework also understands the underlying type ("/person/politician") for the given entity by learning the entity representation with contextual understanding ("barack obama gave a speech to congress"). by using the type and the factual information the framework enhances the comprehension of the focus entity in downstream applications like language modeling. 1 we note that others have explored what kg facts have already been learned by specific, advanced/contemporary lms @xcite . that work utilized a pre-trained bert model and queried what types of kg facts it contains. in addition, our primary goal is not broad, state-of-the-art performance-though we demonstrate that very strong performance is achievable. rather, our goal is to examine what the complementary strengths, and evident limitations, of language modeling techniques for knowledge and entity type representation are. in doing so, we show that our joint framework yields empirical benefits for individual tasks. our models leverage context-independent word embeddings, and we specifically eschew language models pre-trained on web-scale data. 2 our results further suggest that schema-free approaches to knowledge graph construction/embedding and fine grained entity typing should be studied in greater detail, and competitive, if not state-of-theart, performance can be obtained with comparatively simpler, resource-starved language models. this has promising implications for low-resource, few-shot, and/or domain-specific information extraction needs. using publicly available data, our work has four main contributions. (1) it advocates for a languagemodeling based knowledge graph embedding architecture that achieves state-of-the-art performance on knowledge graph completion/fact prediction against comparable methods. (2) it introduces a neural-based technique based on both knowledge graph embedding and language modeling to predict fine-grain entity types, which yields competitive through state-of-the-art performance against comparable methods. (3) it proposes the joint learning of factual information with the underlying entity types in a shared embedding space. (4) it demonstrates that learning a knowledge graph embedding 1 though the entity typing examples here could be interpreted as being hierarchical, our method neither assumes nor requires any type hierarchy. 2 we do not deny that current pre-trained language models can be effective for other language-based tasks beyond language modeling. however, the reason we do not use transformer lms like bert or gpt-2 is because the amount of data they are pre-trained with can make it difficult to (a) fairly compare to previous work (is it the modeling approach, or the underlying, large-scale data at work?), and (b) identify and track the benefits of learning our tasks jointly. model and language model in a shared embedding space are symbiotic, yielding strong kge performance and drastic perplexity improvements. 3 2 background the underlying information in the knowledge bases is difficult to comprehend and manipulate @xcite . a vast number of knowledge graph embeddings techniques have been proposed over the years to mirror the entities and relations in the knowledge graphs. rescal @xcite is one of the first semantic-based embedding technique that captures the latent interaction between the entities and the relation. a model such as rescal can use graph properties to improve the underlying entity and relation representations @xcite @xcite . a more simplified approach is defined in distmult @xcite by restricting the relation matrix to a diagonal matrix. neural tensor network (ntn) @xcite is one such technique that combines the relation specific tensors with head and tail vector representation over non-linear activation function mapped to hidden layer representation. translational methods like transe @xcite use distanced based models to represent entities and the relationships in the same vector space r d . transh @xcite overcomes the shortcomings of transe by modeling the vector representation with relations specific hyperplane. transr @xcite , transd @xcite model the representation similar to transh by having relation specific spaces and decomposing the relation specific projection matrix as a product of two vector representations respectively. recognition of entity types into coarse grain types has been explored by researchers over the past two decades. neural approaches have brought advances in extending the prediction problem from coarse grain entity types to fine-grain entity types. work by @xcite was one of the first attempts in predicting the fine-grain entity types. the work framed the problem as multi-class multilabel classification. this work also led to an important contribution of a labeled dataset figer, widely used as a benchmark dataset in measuring the performance of fine-grain entity type prediction architectures. @xcite introduced the method of automatic fine-grain entity typing by using hierarchical partial label embedding. @xcite introduced a neural fine-grain entity type prediction architecture that uses semantic context with self-attention and handcrafted features to capture semantic context needed for fine-grain type prediction. @xcite showed that analyzing sentences with a pre-trained language model enhanced prediction performance. @xcite introduced a document level context and signifies the importance of mention level attention mechanism along with the sentence-level context in enhancing the performance of fine-grain entity prediction. @xcite enhanced neural fine-grain entity typing by penalizing the cross-entropy loss with hierarchical context loss for the fine-grain type prediction. language modeling has seen great progress in recent times. @xcite pioneered the renewed use of distributed representation for dealing with the dimensionality curse imposed by the statistical methods. their language model used recurrent neural networks for dealing with long sequences of text. @xcite extended the idea of building the recurrent neural networkbased language models with an improved feedback mechanism of backpropagation in time. we are not the first to examine the intersection of knowledge graph embedding and language modeling. @xcite ; @xcite directly embed rdf graphs using languagemodeling based techniques. @xcite and @xcite have more recently leveraged information from a knowledge base to improve language modeling. however, in addition to knowledge graphs and language modeling, we additionally consider fine-grain entity typing. with the success of contextualized vector representations and the availability of large-scale, pretrained language models, there have been a number of efforts aimed at improving the knowledge implicitly contained in word and sentence representations. for example, @xcite introduce comet, which describes a framework to learn and generate rich and diverse common-sense descriptions via language models (e.g., the autoregressive gpt-2). similarly, @xcite and @xcite provide insights into aspects of lm on downstream nlp tasks. while we share the overall goal of improving knowledge representation within language modeling, the short-term goals are dif- word emebddings final state feed forward architecture triplet (leon w johnson, institution, california institute of technology) sigmoid initial state plausibility prediction output from final time step ferent, as we focus on individual facts, rather than traditional background/commonsense knowledge, and demonstrating the complementary nature of kge, entity typing, and lm. ## methodology this section introduces the framework for jointly learning knowledge graph embedding (kge), fine grain entity types (et) and language models (lm). it uses a multi-task learning architecture built over baseline architectures for all three tasks. we begin by introducing lm-inspired knowledge graph embedding and fine grain entity typing architectures; we describe the joint learning architectures in §5. fundamentally, our approach relies on appropriate and select parameter sharing across the kge, et, and lm tasks in order to learn these models jointly. while joint learning or multi-task learning through shared parameters have been examined before for a number of tasks, we argue that this parameter sharing is a very effective way to improve kge, et, and/or lm (for a particular baseline). its simplicity is a core benefit. ## experimental settings the input to the joint learning architectures are the pre-trained glove embedding vectors trained on 840 billion words @xcite . the parameters of the baseline and the joint learning architecture are learned with stochastic gradient descent and adam (kingma and ba, 2014) as a learning rate optimizer. the training of the joint learning networks is performed with alternating optimization. the loss functions of the respective tasks are optimized at each alternate epoch/ interval. the hyper-parameters for training these joint architecture are chosen manually for the bestperforming models on validation sets. data for a direct comparison of the performance as possible, we use previously studied datasets. we evaluate kg triple classification using the standard datasets of wordnet 11 (wn11) and freebase 13 (fb13). wn11 @xcite ) is a publicly available lexical graph of synsets (synonyms). freebase @xcite is a collaborative ontology consisting of factual tuples of entities related to each other through semantic relation. while recent work has advocated for examining variants and other derivatives of these datasets such as fb15k-237 and wn18rr @xcite @xcite , there is a relative lack of previous experimental work on these newer datasets. given space limitations, and in order to compare to the vast majority of previous work, we chose to report on the more common wn11 and fb13. we evaluate fine grain entity type prediction on the well-studied ontonotes @xcite and figer @xcite datasets. the ontonotes dataset used here is a manually curated dataset by @xcite , consisting of 89 different entity types. figer consists of 113 entity types, occuring in sentences from 780k wikipedia articles and 434 news reports. we evaluate the joint kge and entity typing model on wikiauto and wikiman, both introduced by @xcite . wikiauto is curated by distant supervision, with freebase entities and types and sentence descriptions from wikipedia articles. wikiman is a manually curated dataset from wikipedia articles with freebase entities. lastly, we evaluate the joint kge and lm on wikifact @xcite , built using the facts from freebase and wikipedia descriptions. the content of the dataset is limited to film/actor/ from freebase. further the anchor fact defined in the text of the dataset are not used for training the joint model. the description of the entities in the original dataset contain both the summary and the body from wikipedia. the current study is performed by using the description from the summary section defined in the dataset. the joint model is trained and evaluated with the split of 80/10/10 for train, validation and test sets, respectively. metrics kge triple classification is evaluated through accuracy. the entity type model's performance is evaluated based on three common entity typing metrics-strict f1, loose macro f1 and loose micro f1 @xcite -while language modeling is measured by perplexity. previous work as baselines when possible, we directly compare our model's performance to that of previously published work. ## results and discussion this section presents the results of our basic kge, entity typing models, and the joint learning architecture and their comparison to previous methods. the models were trained using either a 16gb v100 or 11gb 2080 ti gpu (single gpu training only). ## conclusion this work proposes a joint learning framework for learning real value representations of words, entities, and relations in a shared embedding space. joint learning of factual representation with contextual understanding shows improvement in the learning of entity types. learning the language model with knowledge graph embedding simultaneously enhances the performance on both modeling tasks. our results suggest that language modeling could accelerate the study of schema-free approaches to both kge and fner, and strong performance can be obtained with comparatively simpler, resourcestarved language models. this has promising implications for low-resource, and few-shot, and/or domain-specific information extraction needs.