KEPLER : A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2021)

Abstract: Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M1 , a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER .

## introduction recent pre-trained language representation models (plms) such as bert @xcite and roberta @xcite learn effective language representation from large-scale unstructured corpora with language modeling objectives and have achieved superior performances on various natural language processing (nlp) tasks. existing plms learn useful linguistic knowledge from unlabeled text @xcite ), but they generally cannot capture the world facts well, which are typically sparse and have complex forms in text @xcite . by contrast, knowledge graphs (kgs) contain extensive structural facts, and knowledge embedding (ke) methods @xcite @xcite can effectively embed them into continuous entity and relation embeddings. these embeddings can not only help with the kg completion but also benefit various nlp applications @xcite @xcite . as shown in figure 1 , textual entity descriptions contain abundant information. intuitively, ke methods can provide factual knowledge for plms, while the informative text data can also benefit ke. inspired by @xcite , we use entity descriptions to bridge the gap between ke and plm, and align the semantic space of text to the symbol space of kgs @xcite . we propose kepler, a unified model for knowledge embedding and pre-trained language representation. we encode the texts figure 1 : an example of a kg with entity descriptions. the figure suggests that descriptions contain abundant information about entities and can help to represent the relational facts between them. and entities into a unified semantic space with the same plm as the encoder, and jointly optimize the ke and the masked language modeling (mlm) objectives. for the ke objective, we encode the entity descriptions as entity embeddings and then train them in the same way as conventional ke methods. for the mlm objective, we follow the approach of existing plms @xcite . kepler has the following strengths: as a plm, (1) kepler is able to integrate factual knowledge into language representation with the supervision from kg by the ke objective. (2) kepler inherits the strong ability of language understanding from plms by the mlm objective. (3) the ke objective enhances the ability of kepler to extract knowledge from text since it requires the model to encode the entities from their corresponding descriptions. (4) kepler can be directly adopted in a wide range of nlp tasks without additional inference overhead compared to conventional plms since we just add new training objectives without modifying model structures. there are also some recent works @xcite @xcite directly integrating fixed entity embeddings into plms to provide external knowledge. however, (1) their entity embeddings are learned by a separate ke model, and thus cannot be easily aligned with the language representation space. (2) they require an entity linker to link the text to the corresponding entities, making them suffer from the error propagation problem. (3) compared to vanilla plms, their sophisticated mechanisms to link and use entity embeddings lead to additional inference overhead. as a ke model, (1) kepler can take full advantage of the abundant information from entity descriptions with the help of the mlm objective. for pre-training and evaluating kepler, we need a kg with (1) large amounts of knowledge facts, (2) aligned entity descriptions, and (3) reasonable inductive-setting data split, which cannot be satisfied by existing ke benchmarks. therefore, we construct wikidata5m, containing about 5m entities, 20m triplets, and aligned entity descriptions from wikipedia. to the best of our knowledge, it is the largest general-domain kg dataset. we also benchmark several classical ke methods and give data splits for both the transductive and the inductive settings to facilitate future research. to summarize, our contribution is three-fold: (1) we propose kepler, a knowledge-enhanced plm by jointly optimizing the ke and mlm objectives, which brings great improvements on a wide range of nlp tasks. ## kepler as shown in figure 2 , kepler implicitly incorporates factual knowledge into language representations by jointly training with two objectives. in this section, we detailedly introduce the encoder structure, the ke and mlm objectives, and how we combine the two as a unified model. ## wikidata5m as shown in section 2, to train kepler, the kg dataset should (1) be large enough, (2) contain high-quality textual descriptions for its entities and relations, and (3) have a reasonable inductive setting, which most existing kg datasets do not provide. thus, based on wikidata foot_0 and english wikipedia, foot_1 we construct wikidata5m, a largescale kg dataset with aligned text descriptions from corresponding wikipedia pages, and also an inductive test set. in the following sections, we first introduce the data collection (section 3.1) and the data split (section 3.2), and then provide the results of representative ke methods on the dataset (section 3.3). ## experiments in this section, we introduce the experiment settings and results of our model on various nlp and ke tasks, along with some analyses on kepler. ## analysis in this section, we analyze the effectiveness and efficiency of kepler with experiments. all the hyperparameters are the same as reported in section 4.1, including models in the ablation study. ## related work pre-training in nlp there has been a long history of pre-training in nlp. early works focus on distributed word representations @xcite @xcite , many of which are often adopted in current models as word embeddings. these we sort the entity mentions in tacred by their corresponding entity frequencies in wikipedia. the ''0%-20%'' setting indicates only keeping the least frequent 20% entity mentions and masking all the other entity mentions (for both training and validation), and so on. the results are averaged over 5 runs. pre-trained embeddings can capture the semantics of words from large-scale corpora and thus benefit nlp applications. peters et al. (2018) push this trend a step forward by using a bidirectional lstm to form contextualized word embeddings (elmo) for richer semantic meanings under different circumstances. apart from word embeddings, there is another trend exploring pre-trained language models. @xcite propose to train an auto-encoder on unlabeled textual data and then fine-tune it on downstream tasks. @xcite propose a universal language model (ulmfit). with the powerful transformer architecture @xcite , @xcite demonstrate an effective pre-trained generative model (gpt). later, @xcite release a pre-trained deep bidirectional encoder representation from transformers (bert), achieving state-of-the-art performance on a wide range of nlp benchmarks. after bert, similar plms spring up recently. @xcite propose a permutation language model (xlnet). later, @xcite . other works explore how to add more tasks @xcite and more parameters @xcite to plms. knowledge-enhanced plms recently, many works have investigated how to incorporate knowledge into plms. mtb @xcite takes a straightforward ''matching the blank'' pre-training objective to help the relation classification task. ernie @xcite identifies entity mentions in text and links pre-processed knowledge embeddings to the corresponding positions, which shows improvements on several nlp benchmarks. with a similar idea as ernie, knowbert @xcite incorporates an integrated entity linker in their model and adopts end-to-end training. besides, @xcite and @xcite utilize relations between entities inside one sentence to train better generation models. @xcite adopt entity replacement knowledge learning for improving entity-related tasks. some contemporaneous or following works try to inject factual knowledge into plms in different ways. e-bert @xcite aligns entity embeddings with word embeddings and then directly adds the aligned embeddings into bert to avoid additional pre-training. k-adapter @xcite injects knowledge with additional neural adapters to support continuous learning. knowledge embedding ke methods have been extensively studied. conventional ke models define different scoring functions for relational triplets. for example, transe @xcite treats tail entities as translations of head entities and uses l 1 -norm or l 2 -norm to score triplets, while distmult @xcite uses matrix multiplications and complex @xcite adopts complex operations based on it. rotate @xcite combines the advantages of both of them. inductive embedding above ke methods learn entity embeddings only from kg and are inherently transductive, while some works @xcite @xcite @xcite incorporate textual metadata such as entity names or descriptions to enhance the ke methods and hence can do inductive ke to some extent. besides kg, it is also common for general inductive graph embedding methods @xcite to utilize additional node features like text attributes, degrees, and so on. kepler follows this line of studies and takes full advantage of textual information with an effective plm. @xcite and wang et al. (2019c) perform inductive ke by aggregating the trained embeddings of the known neighboring nodes with graph neural networks, and thus do not need additional features. but these methods require the unseen nodes to be surrounded by known nodes and cannot embed new (sub)graphs. we leave how to develop kepler to do fully inductive ke without additional features as future work. ## conclusion and future work in this paper, we propose kepler, a simple but effective unified model for knowledge embedding and pre-trained language representation. we train kepler with both the ke and mlm objectives to align the factual knowledge and language representation into the same semantic space, and experimental results on extensive tasks demonstrate its effectiveness on both nlp and ke applications. besides, we propose wikidata5m, a large-scale kg dataset to facilitate future research. in the future, we will (1) explore advanced ways for more smoothly unifying the two semantic space, including different ke forms and different training objectives, and (2) investigate better knowledge probing methods for plms to shed light on knowledge-integrating mechanisms.