Sequence-to-Sequence Knowledge Graph Completion and Question Answering (2022)

Abstract: Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. For downstream tasks these atomic entity representations often need to be integrated into a multi stage pipeline, limiting their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and incomplete KG question answering. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces the model size up to 98% compared to conventional KGE models while keeping inference time tractable. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning.

## introduction a knowledge graph (kg) is a multi-relational graph where the nodes are entities from the real world (e.g. barack obama, united states) and the named edges represent the relationships between them (e.g. barack obama -born in -united states). kgs can be either domain-specific such as wikimovies @xcite or public, crossdomain kgs encoding common knowledge such as wikidata and dbpedia @xcite . these graph-structured databases play an important role 1 resources are available at https://github.com/ apoorvumang/kgt5 in knowledge-intensive applications including web search, question answering and recommendation systems @xcite . most real-world knowledge graphs are incomplete. however, some missing facts can be inferred using existing facts in the kg @xcite . this task termed knowledge graph completion (kgc) foot_0 has become a popular area of research in recent years @xcite and is often approached using knowledge graph embedding (kge) models. kge models represent each entity and relation of the kg by a dense vector embedding. using these embeddings the model is trained to distinguish correct from incorrect facts. one of the main downstream applications of kges is question answering over incomplete kgs (kgqa) @xcite . taking into account the large size of real world kgs (wikidata contains ≈90m entities) and the applicability to downstream tasks, kge models should fulfill the following desiderata: (i) scalability -i.e. have model size and inference time independent of the number of entities (ii) quality -reach good empirical performance (iii) versatility -be applicable for multiple tasks such as @xcite simplicity -consist of a single module with a standard architecture and training pipeline. traditional kge models fulfill quality and simplicity. they build upon a simple architecture and reach a high quality in terms of kgc. however, as they create a unique embedding per entity/relation, they scale linearly with the number of entities in the graph, both in model size and inference time, and offer limited versatility. methods such as dkrl @xcite and kepler @xcite attempt to tackle the scalability issue using compositional embeddings. however, they fail to achieve quality comparable to conventional kges. kg-bert @xcite utilizes pretrained bert for link prediction and holds pofigure 1 : overview of our method kgt5. kgt5 is first trained on the link prediction task (predicting head/tail entities, given tail/head and relation). for question answering, the same model is further finetuned using qa pairs. tential in terms of versatility as it is applicable to downstream nlp tasks. however, it is not scalable due to its underlying cross-encoder. 3 qa methods which leverage kges outperform traditional kgqa approaches on incomplete kgs, but combining kges with the qa pipeline is a non-trivial task; models that attempt to do this often work on only limited query types @xcite @xcite ) or require multistage training and inference pipelines @xcite . here, in order to achieve quality, these models have sacrificed versatility and simplicity. a comparison of approaches in terms of desiderata is summarized in tab. 9 in the appendix. our paper shows that all of these desiderata can be fulfilled by a simple sequence-to-sequence (seq2seq) model. to this end, we pose kg link prediction as a seq2seq task and train an encoderdecoder transformer model @xcite on this task. we then use this model pretrained for link prediction and further finetune it for question answering; while finetuning for qa, we regularize with the link prediction objective. this simple but powerful approach, which we call kgt5, is visualised in fig. 1 . with such a unified seq2seq approach we achieve (i) scalability -by using compositional entity representations and autoregressive decoding (rather than scoring all entities) for inference (ii) quality -we obtain state-of-the-art performance on two tasks (iii) versatility -the same model can be used for both kgc and kgqa on multiple datasets, and (iv) simplicity -we obtain all results using an off-the-shelf model with no task or dataset-specific hyperparameter tuning. in summary, we make the following contributions: • we show that kg link prediction and question answering can be treated as sequence-to-sequence tasks and tackled successfully with a single encoder-decoder transformer (with the same architecture as t5-small @xcite ). kgt5, we reduce model size for kg link prediction up to 98% while outperforming conventional kges on a dataset with 90m entities. ## background & related work given a set of entities e and a set of relations r, a knowledge graph k ⊆ e × r × e is a collection of subject-predicate-object (s, p, o) triples. link prediction is the task of predicting missing triples in k by answering queries of the form of (s, p, ?) and (?, p, o). this is typically accomplished using knowledge graph embedding (kge) models. conventional kges assign an embedding vector to each entity and relation in the kg. they model the plausibility of (s, p, o) triples via model specific scoring functions f (e s , e p , e o ) using the subject (e s ), predicate (e p ) and object (e o ) specific embeddings. once trained, these embeddings are used for downstream tasks such as question answering. knowledge graph question answering (kgqa) is the task of answering a natural language question using a kg as source of knowledge. the questions can be either simple factual questions that require single fact retrieval (e.g. which languages are spoken in india?), or they can be complex questions that require reasoning over multiple facts in the kg (e.g. what are the genres of movies, in which leonardo dicaprio was leading actor?). kges can be utilized to perform kgqa when the background kgs are incomplete. in the next few sections we will go into more detail about existing work on kges and kgqa. ## the kgt5 model we pose both knowledge graph link prediction and question answering as sequence-to-sequence (seq2seq) tasks. we then train a simple encoderdecoder transformer -that has the same architecture as t5-small @xcite but without the pretrained weights -on these tasks. while training for question answering, we regularize with the link prediction objective. this method, which we call kgt5, results in a scalable kg link prediction model with vastly fewer parameters than conventional kge models for large kgs. this approach also confers simplicity and versatility to the model, whereby it can be easily adapted to kgqa on any dataset regardless of question complexity. posing kg link prediction as a seq2seq task requires textual representations of entities and relations, and a verbalization scheme to convert link prediction queries to textual queries; these are detailed in §3.1. the link prediction training procedure is explained in §3.2 and inference in §3.3. the kgqa finetuning and inference pipeline is explained in §3.4. ## experimental study we investigate whether kgt5-i.e. a simple seq2seq transformer model-can be jointly trained to perform both knowledge graph link prediction as well as question answering. hereby, we first describe the used datasets ( §4.1), the baselines we compared to ( §4.2) and the experimental setup ( §4.3). the results of our experiments are analysed in §4.4- §4.8. before going into detail, we summarize our key findings: 1. for link prediction on large kgs, the text-based approach of kgt5 reduces model size to comparable kge models by up to 98% and reaches or outperforms current state-of-the-art. 2. on the task of kgqa over incomplete kgs, our simple seq2seq approach obtains better results than the current state-of-the-art across multiple datasets. 3. kg link prediction training might be more beneficial than language modeling pretraining on knowledge intensive tasks such as kgqa. 4. although kgt5 is good at generalizing to unseen facts, it is rather poor at memorizing facts. this problem can be alleviated, if needed, by using an ensemble of kgt5 and conventional link prediction or kgqa systems. ## conclusion and future work we have shown that kg link prediction and question answering can be treated as seq2seq tasks and tackled successfully with a single encoder-decoder transformer model. we did this by training a transformer model with the same architecture as t5small on the link prediction task, and then finetuning it on the qa task. this simple but powerful approach, which we call kgt5, performed competitively with the state-of-the-art methods for kg completion on large kgs while using upto 98% fewer parameters. on the task of kgqa on incomplete kgs, we found that our unified approach outperformed baselines on multiple largescale benchmark datasets. additionally, we compared language modeling pretraining with kg link prediction training and found that for knowledgeintensive tasks such as kgqa, link prediction training could be more beneficial. one promising direction for future exploration would be to see whether kg link prediction training could be considered as an additional pretraining objective when training large seq2seq models. furthermore, the impact of model size, and whether larger transformer models can indeed store more relational information should be investigated.