Pretrain- KGE : Learning Knowledge Representation from Pretrained Language Models (2020)

Abstract: Conventional knowledge graph embedding (KGE) often suffers from limited knowledge representation, leading to performance degradation especially on the low-resource problem. To remedy this, we propose to enrich knowledge representation via pretrained language models by leveraging world knowledge from pretrained models. Specifically, we present a universal training framework named Pretrain-KGE consisting of three phases: semantic-based fine-tuning phase, knowledge extracting phase and KGE training phase. Extensive experiments show that our proposed Pretrain-KGE can improve results over KGE models, especially on solving the low-resource problem.

## introduction knowledge graphs (kgs) constitute an effective access to world knowledge for a wide variety of nlp tasks, such as entity linking @xcite , information retrieval @xcite , question answering @xcite and recommendation system @xcite . a typical kg such as freebase @xcite and word-net @xcite , consists of a set of triplets in the form of (h, r, t) with the head entity h and the tail entity t as nodes and relation r as edges in the graph. a triplet represents the relation between two entities, e.g., (steve jobs, founded, apple inc.). to learn effective representation of entities and relations in the graph, knowledge graph embedding (kge) models are one of prominent approaches @xcite @xcite @xcite @xcite @xcite . however, traditional kge models often suffer from limited knowledge representation due to the sparse and noisy dataset annotations. it leads to performance degradation, especially on the lowresource problem. to address this issue, we propose to enrich knowledge representation via pretrained language models (i.e., bert @xcite ) given a semantic description of entities and relations. we propose to incorporate world knowledge from bert to the entity and the relation representation. although simply fine-tuning bert can enrich the knowledge representation, it suffers from learning inadequate structure information observed in training triplets, which we have demonstrated when we analyze the rationality of the kge-training phase. we propose a model-agnostic training framework for learning knowledge graph embedding consisting of three phases: semantic-based fine-tuning phase, knowledge extracting phase and kge training phase (see fig. 1 ). during the semantic-based fine-tuning phase, we learn knowledge representation via bert given the semantic description of entities and relations as the input sequence. in this way, we incorporate world knowledge from bert into the knowledge representation. then during the knowledge extracting phase, we extract the entity and the relation representations encoded by bert and inject them into embeddings of a kge model. finally, during the kge training phase, we train the kge model to learn adequate structure information of dataset, while reserving partial knowledge from bert to learn better knowledge graph embedding. extensive experiments show that our proposed pretrain-kge can improve performance over kge models on four benchmark kg datasets. further analysis and visualization of the knowledge learning process demonstrate that our method can enrich knowledge representation via pretrained language models through the training framework. 2 related work kge models can be roughly divided into translational models and semantic matching models according to the score function @xcite . translational models consider the relation between the head and tail entity as a translation between the two entity embeddings, such as transe @xcite , transh @xcite , transr @xcite , transd @xcite , rotate @xcite , and toruse @xcite ; while semantic matching models define a score function to match latent semantics of the head, tail entity and the relation, such as, rescal @xcite , dist-mult @xcite , simple (kazemi and poole, 2018), complex @xcite and quate @xcite . quate @xcite is the recent state-of-the-art kge model, which represents entities as hypercomplex-valued embeddings and models relations as rotations in the quaternion space. in a knowledge graph dataset, the names of each entity and relation are provided as the semantic description of entities and relations. recent works also leverage semantic description to enrich knowledge representation but ignore contextual information of the semantic description @xcite @xcite @xcite @xcite . instead, our method exploits world information via pretrained models. recent approaches to modeling language representations offer significant improvements over embeddings, such as pretrained deep contextualized language models @xcite @xcite . kg-bert @xcite first utilizes bert @xcite for knowledge graph completion, which treats triplets in knowledge graphs as textual sequences. however, kg-bert does not extract knowledge representations from bert and thus cannot provide entity or relation embeddings. in this work, we leverage world knowledge from bert to learn better knowledge representation of entities and relations given semantic description. ## analysis in this section, we evaluate our pretrain-kge on the low-resource problem and further verify the rationality of our training framework. ## conclusion we propose pretrain-kge, an efficient pretraining technique for learning knowledge graph embedding. pretrain-kge is a universal training framework that can be applied to any kge model. it learns knowledge representation via pretrained language models and incorporates world knowledge from the pretrained model into the entity and the relation embedding. extensive experimental results demonstrate consistent improvements over kge models across multiple benchmark datasets. the knowledge incorporation introduced in pretrain-kge alleviates the low-resource problem and we justify our three-phase training framework through an analysis of the knowledge learning process.