From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer
ABSTRACT
Knowledge graph completion aims to address the problem of extending a KG with missing triples.
In this paper, we provide an approach GenKGC , which converts knowledge graph completion to sequence-to-sequence generation task with the pre-trained language model.
We further introduce relation-guided demonstration and entity-aware hierarchical decoding for better representation learning and fast inference.
Experimental results on three datasets show that our approach can obtain better or comparable performance than baselines and achieve faster inference speed compared with previous methods with pre-trained language models.
We also release a new large-scale Chinese knowledge graph dataset
OpenBG500 for research purpose 1 .
CONCEPTS
→ Knowledge representation
ACM Reference Format: Xin Xie, Ningyu Zhang, Zhoubo Li, Shumin Deng, Hui Chen, Feiyu Xiong,
Mosha Chen, and Huajun Chen.
From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer.
In Companion Proceedings of the Web Conference 2022 (WWW ’22 Companion), April 25– 29, 2022, Virtual Event, Lyon, France.
ACM, New York, NY, USA, 4 pages.
https://doi.org/10.1145/3487553. ∗ Corresponding author.
1 Code and datasets are available in https://github.com/zjunlp/PromptKG/tree/main/ research/GenKGC
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than ACM mustbehonored.
Abstractingwithcreditispermitted.
Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/ora fee.
Request permissions from permissions@acm.org.
WWW ’22 Companion, April 25–29, 2022, Virtual Event, Lyon, France © 2022 Association for Computing Machinery.
ACM ISBN [-978-1-4503-XXXX-X/18/06... $15.
-]{+978-1-4503-9130-6/22/04...$15.
+}
https://doi.org/10.1145/3487553.
INTRODUCTION
Knowledge Graphs (KGs) treat the knowledge in the real world as fact triples in the form of <subject, predicate, object> , abridged as ( 𝑠,𝑝,𝑜 ) , where 𝑠 and 𝑜 denote entities and 𝑝 are the relationsbetweenentities, whichcanbenefitawiderangeofknowledgeintensive tasks.
Knowledge graph completion (KGC) aims to complete the knowledge graph by predicting the missing triples.
In this paper, we mainly target link prediction task for KGC based on the powerful pre-trained language models.
Most previous KG completion methods, such as TransE [2], ComplEx [11], and RotatE [9], are knowledge embedding techniques that embed the entities and relations into a vector space and then obtain the predicted triples by leveraging pre-defined scoring functions to those vectors.
Recently, some textual encoding methods (e.g., KG-BERT [14]) have been proposed, which utilize the pretrained language model to encode triples and output the score for each candidate.
Obviously, most previous approaches leverage the discrimination paradigm with a pre-defined scoring function to knowledge embeddings.
However, such a discrimination strategy has to costly scoring of all possible triples in inference and suffer from the instability of negative sampling.
Moreover, those dense knowledge embedding approaches (e.g., TransE) ignore the finegrained interactions between entities and relations and have to allocate a large memory footprint for the large-scale real-world knowledge graph.
Therefore, it is intuitive to find a new technical solution for knowledge graph completion.
In this paper, we take the first step to model the knowledge graph completion with sequence to sequence generation and propose a novel approach GenKGC .
To be specific, we represent entities and relations as input sequences and utilize the pre-trained language model to generate target entities.
Following GPT-3’s naive "in-context learning" paradigm, in which the model can learn correct output answers by concatenating the selected samples relevant to the input, we propose relation-guided demonstration by adding triples of the same relation.
Moreover, during generation, we propose entity-aware hierarchical decoding to reduce the time complexity of generation.
Experimental results on two datasets
WN18RR, FB15k-237 and a newly released large-scale Chinese KG dataset OpenBG500 demonstrate the effectiveness of the proposed approach.
The contributions of our work are as follows:
WWW
Pre-trained Encoder
Pre-trained Decoder
Michael Chabon studied at
Triple :< Michael Chabon ,study at, ?
> [Univ]
Type Decode
Entity Decode demonstration <bos>
[Univ] <eos> <bos> [LOC] [ORG]
[Univ] ...
Irvine
Davis ...
UC
San ...
<eos> <eos> <eos>
Harvard University
UC UC
Irvine
D
Obama studied at Harvard.
Yau studied at, UC Berkeley.
Irvine
• We convert link prediction to sequence to sequence generation and propose GenKGC , which can reduce the inference time while maintaining the performance.
• We propose relation-guided demonstration and entity-aware hierarchical decoding, which can better represent entities and relations and reduce the time complexity of generation.
• We report results on two datasets and release a new largescale KG dataset, OpenBG500, for research purposes.
METHOD 2. Link Prediction as Seq2Seq Generation
Knowledge graph is defined with entity categories and entity descriptions as a tuple G =
(E , R , T , C , D) , where E represents a set of entities, R represents relation types, T refers to a set of triples , C refers to the entity categories and D refers to the entity descriptions.
For each triple 𝑡 ∈ T , there is the form ( 𝑒 𝑖 ,𝑟 𝑗 ,𝑒 𝑘 ) where 𝑒 𝑖 ,𝑒 𝑘 ∈ E is the head and tail entity respectively.
For each entity 𝑒 𝑖 ∈ E , there exists category 𝑐 𝑖 to define the 𝑒 𝑖 and a text 𝑑 𝑖 to describe 𝑒 𝑖 .
To complete missing triples in KGs, link prediction aims at predicting the tail entity given the head entity and the query relation, denoted by ( 𝑒 𝑖 ,𝑟, ? ) .
In this paper, we utilize the standard encoder-decoder architecture for sequence-to-sequence generation.
Note that we regard each entity and relation as the sequence of tokens.
Concretely, we follow [14] to use plain text to represent the entities and relations instead of unique embedding to bridge the gap between the triples in the knowledge graph and pre-trained language models.
To be specific, given a triple with tail entity missing ( 𝑒 𝑖 ,𝑟 𝑗 , ? ) , we obtain the description 𝑑 𝑒 𝑖 𝑑𝑟 𝑗 of 𝑒 𝑖 and 𝑟 𝑗 respectively, and concatenate them together.
For example, as shown in Figure 1, "Michael Chabon studied at" (head entity, relation) is the major part of the input sequence, and "UC, Irvine" (target entity) is part of the output sequence.
Thus, we have the input sequence of < 𝑒 𝑖 ,𝑟 > pair and generate the output sequence of 𝑒 𝑘 .
We leverage BART for model training and inference.
[Table]
2. Relation-guided Demonstration
Inspired by prompt-tuning [18], we propose relation-guided demonstration for the encoder.
Note that there exist long-tailed distributions in the KGs, for example, there exist only 37 instances of the relation film/type_of_appearance in the FB15k-237 dataset.
Previous study [18] illustrates that concatenating randomly sampled instances as demonstrations can yield better few-shot performance.
Thus, we construct relation-guided demonstration examples { 𝑟 in ,𝑡 train } .
To be specific, we sample those demonstrations with the guidance of relation 𝑟 𝑗 , which consists of several triples with the same relation of input from training set.
Formally, we have the final input sequence as:
[Equation]
2. Entity-aware Hierarchical Decoding
In the vanilla decoding setting, we have to enumerate all entities in the E and then sort them by the score function.
However, this process can be time-consuming, as shown in Table 1 when E is very large, it is costly scoring of all possible triples.
In our approach, we follow [3] to use the Beam Search to obtain top- 𝑘 entities in the
E ( 𝑘 is the hyperparameter of beam size).
Intuitively, there is no need for negative sampling as we directly optimize by predicting
[Table]
the correct entity in decoding.
To be more specific, given with a triple with tail or head entity missing ( 𝑒 𝑖 ,𝑟 𝑗 , ? ) , GenKGC rank each 𝑒 ∈ E by computing a score with an autoregressive formulation:
[Equation]
where 𝑧 is the set of | 𝐶 | tokens in the category 𝑐 , 𝑦 is the set of 𝑁 tokens in the textual representation of 𝑒 .
Since KG contains rich semantic information such as entity types, it is intuitive to constrain the decoding for fast inference.
We sample the type category with the lowest frequency of occurrence in the training set to constrain the entity decoding since it is challenging to discriminate those low-frequent entities.
Then, we add special tokens as types in the vocabulary of pre-trained language model to constrain the decoding.
To ensure that the generated entities are inside the entity candidate set, we construct a prefix tree (trie tree) to decode our entity name as shown in Figure 1.
Similar to the ordinary sequence-to-sequence model, we optimize GenKGC using a standard seq2seq objective function:
[Equation]
EXPERIMENT
Datasets.
We evaluate our method on FB15k-237 [10], WN18RR
[5], which are widely used in the link prediction literature, and a new real-world large-scale Chinese KG dataset OpenBG500 2 .
In FB15k-237, descriptions of entities are obtained from the introduction section of the Wikipedia page of each entity.
In WN18RR, each entity corresponds to a word sense, and description is the word definition.
In OpenBG500, the descriptions of entities and relations come from the e-commerce description page.
More details about datasets are listed in Table 3.
[Table]
Setting.
We adopt BART-base as our backbone for comparison with other BERT-based KGC methods like KG-BERT.
Following
StAR [12], we choose the two kinds of KGE methods as our baseline models, which can be classified as graph embedding approach and textual encoding approach.
Grid search is used over the three datasets and the results are reported on the test set with hyperparameters of the best performance determined by the dev set.
Metrics.
We evaluate the test set of triples T ′ disjoint from the set of training triples T .
For inference on a test triple ( 𝑒 𝑖 ,𝑟 𝑗 ,𝑒 𝑘 ) , we make sure predict the entity in the candidate set E .
We use the metrics of hits@1, hits@3 and hits@10.
Comparison with Discrimination Method.
From Table 2, we notice that GenKGC achieves better performance than KG-BERT [14] across all datasets and maintains high speed during inference.
The translation-based method like TransE, which treats entities and relations as dense vectors in the same space, will face the memory explosion problem.
For the memory cost, TransE has to consume 260M parameters to store the entities and relations for OpenBG with more than 260k entities, while pre-trained models (BERT or
BART) only utilize 110M parameters, which demonstrates the memory efficiency of our approach.
Note that this problem will be more severe when the entities become more numerous because the space complexity is 𝑂 ( 𝑛 ) .
For the inference speed, KG-BERT encodes the relational triples with the pre-trained language model and ranks all candidate triples with correct and wrong entities for inference.
When the candidate entities set is huge, it is time-consuming for inference; for example, as shown in Table 1, KG-BERT takes about 100100s to the scoring of all possible triples given a single (entity,
[Table]
relation) pair.
While our method only needs to generate the top- 𝑘 entities with entity-aware hierarchical decoding, which reduces lots of computing resources .
3. Case Study
For different decoding strategies we conduct case study to analyze the result.
For GenKGC w/o hierarchical decoding, we utilize the normal beam search to decode the text name of the missing entity.
From Table 4, we observe that GenKGC obtain better entity generation results while in normal beam search, the model may stop early at correct but not precise enough answer.
We argue that this is caused by the bias of the pre-trained language model (e.g., common token bias) since high-frequent tokens will lead the pre-trained language model to be biased toward certain answers.
Our entityaware hierarchical decoding can constrain the decoding process and mitigate the bias effect caused by pre-trained language models.
RELATED WORK
Knowledge Graph Embedding Models.
There are lots of methods of KGC are based on knowledge graph embeddings (KGE), which generally leverage an embedding vector in the continuous embedding space to represent the entity and the relation in KG [17].
One kind of KGE methods is translation-based , such as TransE [2], ConE [20], TotatE
[9], which consider relations as the mapping function between entities.
The other kind of KGE method is semantic matching models, where they get the semantic similarity by using the multi-linear or bilinear product.
Pre-trained Language Models for KGC.
Recently, since pre-trained language models, such as BERT [6], have shown significant improvement on several natural language processing tasks, several works use the transformer-based models to tackle the KGC problems [15, 16, 19].
KG-BERT [14] first propose to use BERT for KGC by seeing a triple as a sequence and converts KGC into a sequence classification problem with the binary cross-entropy object.
[7] proposes to use a transformer encoder-decoder model that takes plain text as input and output a structured triple of the information hide in it.
CONCLUSION
In this paper, we propose GenKGC , which can reach comparable results while reducing inference and training cost in link prediction with pre-trained models.
Experimental results on three benchmark datasets demonstrate the effectiveness of our approach, especially in inference time.
ACKNOWLEDGMENTS
This work is funded by NSFC91846204/NSFCU19B2027, National
Key R&D Program of China (Funding No.SQ2018YFC000004), ZhejiangProvincialNaturalScienceFoundationofChina(No.
LGG22F030011),
Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G).