Joint Language Semantic and Structure Embedding for Knowledge Graph Completion (2022)

Abstract: The task of completing knowledge triplets has broad downstream applications. Both structural and semantic information plays an important role in knowledge graph completion. Unlike previous approaches that rely on either the structures or semantics of the knowledge graphs, we propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models with respect to a probabilistic structured loss, where the forward pass of the language models captures semantics and the loss reconstructs structures. Our extensive experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method. We also show that our method can significantly improve the performance in a low-resource regime, thanks to the better use of semantics. The code and datasets are available at https://github.com/pkusjh/LASS .

## introduction knowledge graphs (kg), such as wikidata and freebase @xcite , consist of factual triplets. kgs have been useful resources for both humans and machines. a triplet in the form of (head entity, relation, tail entity), where the relation involves both head and tail entities, has been used in a great variety of applications, such as question answering @xcite and web search @xcite . incompleteness has been a longstanding issue in kgs @xcite , impeding their wider adoption in real-world applications. kg completion aims to predict a missing entity or relation of a factual triplet. structural patterns in the existing triplets are useful to predict the missing elements @xcite . for example, a composition pattern can be learned to predict the relation grandmother_of based on two consecutive mother_of relations. besides the structure information, semantic relatedness between entities and relations is also critical to infer entities or relations with similar meanings @xcite @xcite . for example, if a relationship ceo_of holds between two entities, the relation employee_of also holds. there are two kinds of kg completion approaches that fall into different learning paradigms. first, the structure-based approaches treat entities and relations as nodes and edges, and use graph embedding methods to learn their representations. second, the semantic-based approaches encode the text description of entities and relations via language models. while both structures and semantics are vital to kg completion, it is non-trivial for existing methods to process both structural and semantic information. in this paper, we propose lass, a joint language semantic and structure embedding for knowledge graph completion, which incorporates both semantics and structures in a kg triplet. lass embeds a triplet into a vector space by fine-tuning pre-trained language models (lm) with respect to a structured loss. lass involves both semantic embedding and structure embedding. the semantic embedding captures the semantics of the triplet, which corresponds to the forward pass of a pre-trained lm over the natural language description of the triplet. the structure embedding aims to reconstruct the structures in the semantic embedding, which corresponds to optimizing a probabilistic structured loss via the backpropagation of the lm. intuitively, the structured loss treats the relationship between two entities as a translation between embeddings of the entities. lass outperforms the existing approaches on a collection of kg completion benchmarks. we further evaluate lass in low-resource settings and find that it is more data-efficient than other methods. the reason is that our method ex- figure 1: overview of lass. lass maps a knowledge triplet (head entity, relation, tail entity), in short (h, r, t), to the corresponding embedding vectors, h, r, @xmath0. lass embeds kgs for kg completion via fine-tuning pre-trained language models (lm) w.r.t. a probabilistic structured loss, where the forward pass of the lms captures semantics and the loss reconstructs structures. in particular, lass consists of semantic embedding and structure embedding. the semantic embedding (leftmost arrow) is generated by a forward pass of the lms followed by a pooling layer over the natural language description of a triplet. the contributions are the following: the results suggest that capturing both semantics and structures is critical to understand the kgs. the findings are beneficial to many downstream knowledge-driven applications. ## lass we introduce lass to embed both semantics and structures of knowledge graphs (kg) with natural language. as shown in figure 1 , lass incorporates two embeddings: semantic embedding and structure embedding. the semantic embedding captures the semantics in the natural language description of the kg triplets. the structure embedding further reconstructs the structure information of the kgs from the semantic embedding. lass embeds kg in a vector space by fine-tuning a pre-trained language model (lm) w.r.t. a structured loss, where the forward pass performs semantic embedding and the optimization of structured loss conducts structure embedding. ## conclusion we propose a new embedding method that leverages both semantics and structures of the knowledge graphs for the task of knowledge graph completion, and offers additional benefits in lowresource settings. the method maps a knowledge graph triplet to an embedding space via fine-tuning language models, where the forward pass captures semantics and the loss reconstructs structures. our method has shown significant improvements on knowledge graph completion benchmarks. the implementation has made no modifications to the language model architectures. the results suggest that the learned embeddings are generally useful in downstream knowledge-driven applications, and potentially useful for more natural language understanding tasks. we hope our results will foster further research in this direction.