Knowledge Is Flat: A S eq2 S eq Generative Framework for Various Knowledge Graph Completion (2022)

Abstract: Knowledge Graph Completion (KGC) has been recently extended to multiple knowledge graph (KG) structures, initiating new research directions, e.g. static KGC, temporal KGC and few-shot KGC. Previous works often design KGC models closely coupled with specific graph structures, which inevitably results in two drawbacks: 1) structure-specific KGC models are mutually incompatible; 2) existing KGC methods are not adaptable to emerging KGs. In this paper, we propose KG-S2S, a Seq2Seq generative framework that could tackle different verbalizable graph structures by unifying the representation of KG facts into “flat” text, regardless of their original form. To remedy the KG structure information loss from the “flat” text, we further improve the input representations of entities and relations, and the inference algorithm in KG-S2S. Experiments on five benchmarks show that KG-S2S outperforms many competitive baselines, setting new state-of-the-art performance. Finally, we analyze KG-S2S’s ability on the different relations and the Non-entity Generations.

## introduction knowledge graph completion (kgc) has been a fundamental task to discover unobserved facts from various knowledge graph (kg) structures, including static kgc (skgc), temporal kgc (tkgc) and few-shot kgc (fkgc) @xcite . as shown in figure 1 , tkgc (in orange) contains temporal facts with timestamps, while fkgc (in green) predicts the facts with relations that only have limited or zero training instances. typically, the solutions for kgc are graphbased, i.e., treating entities and relations as nodes and linkages. the training and inference of skgc models rely on various transitional relations over graph paths @xcite @xcite . tkgc and fkgc methods are further integrated with non-trivial components or learning paradigms to handle the extra temporal information or training requirements. concretely, tkgc models @xcite @xcite either construct temporal-specific sub-kg or add additional temporal embeddings into existing skgc methods. fkgc models apply the additional training scheme (e.g., meta-learning) between the frequent relations and the infrequent ones to the skgc models @xcite . such a methodological discrepancy leads to a great maintenance cost and being inadaptable to emerging knowledge queries, ingestion, and presents. naturally, a research question has been raised: can we adapt the different forms of kg facts and solve these kgc tasks in a unified framework? recently, seq2seq pre-trained language models (plm) have shown state-of-the-art performances and high technical homogeneity when dealing with different nlp tasks. albeit having heterogeneous input and output, the seq2seq plms covert those tasks into "text-to-text" format, taking the text as inputs and producing another text as outputs @xcite . in addition, plms have embedded massive real-world knowledge from the pre-training @xcite , which is potentially beneficial for the kgc tasks, especially in the data-sparsity scenarios. inspired by this, we propose kg-s2s, a simple yet effective seq2seq plm framework adaptable to various kg structures. given a kg query, kg-s2s directly generates the target entity text using the common plm fine-tuning practices. firstly, to remedy the kg structure information loss caused by the naïve "text-to-text" format, we improve kg-s2s via 1) the input representations of entities and relations using entity description, soft prompt and seq2seq dropout; 2) the constrained inference algorithm empowered by the prefix constraints; secondly, we treat all the kg elements (i.e., entity, relation and timestamp) as "flat" text (figure 1 ) which enables kg-s2s to i) handle various verbalizable knowledge graph structures; ii) generate non-entity text and find novel entities for kgs. we make several improvements on the preliminary attempts of concurrent works @xcite using seq2seq for kgc. our model adds special treatments to input entity/relation textual representation. this helps to better capture subtle yet key tokens and facilitate the ability to ingest other graph structures. we conduct experiments on wn18rr, fb15k-237 and fb15k-237n for skgc, icews14 for tkgc and nell-one for fkgc. kg-s2s outperforms several competitive baseline models, including graph-based and plm-based models, and sets new state-of-the-art performance on all three settings. we conduct ablation studies to show the effectiveness of the proposed components, compare kg-s2s with graph-based kgc models at the relation level and finally showcase the non-entity generation from kg-s2s to present its potential in producing novel knowledge triples. static kgc early skgc models assign trainable embeddings to each entity and relations @xcite . a score function is proposed to evaluate the scores of triples with these embedding. these models learn structural information of a knowledge graph, regardless of the textual information of the entities and relations. recently, yao et al. (2019 temporal kgc many tkgc models incorporate additional time-specific parameters upon existing kgc methods. @xcite , based on @xcite , represents each timestamp with independent embeddings. @xcite resembles @xcite , regarding timestamps as hyperplanes for entities to project. @xcite considers the score of each triple as canonical decomposition of order 4 tensors in complex domain. @xcite suggests learning dynamic embeddings for entity and relations, transforming part of the embedding with sinusoidal activation of learned frequencies. @xcite proposes a systematic framework to improve existing temporal embedding models. few-shot kgc for one-shot learning on relations, @xcite attempts to seek a matching metric that can be used to discover similar triples given one reference triple. chen et al. (2019) ## proposed method this section first formulates knowledge graph completion tasks in sec. 3.1, then discusses our proposed kg-s2s method in sec. 3.2, 3.3 and 3.4. figure 2 shows the overview of kg-s2s. ## conclusion and future work in this paper, we present kg-s2s for various knowledge graph completion tasks. by converting different kinds of kg structures into "text-totext" format, kg-s2s can directly produce the text of target predicted entities. experimental results demonstrate that kg-s2s outperforms competitive baseline models in various kgc settings. in the future, we would explore extending kg-s2s to other seq2seq plms, such as bart @xcite and mass @xcite . in addition, it is interesting to combine kg-s2s with other knowledge-intensive nlp tasks, such as conversation recommendation @xcite and commonsense generation @xcite in the seq2seq framework, and see if the kg knowledge could benefit these downstream tasks. yukun zuo, quan fang, shengsheng qian, xiaorui zhang, and changsheng xu. 2018. representation learning of knowledge graphs with entity attributes and multimedia descriptions. in fourth ieee international conference on multimedia big data, bigmm 2018, xi'an, china, september 13-16, 2018, pages 1-5. ieee.