Text Augmented Open Knowledge Graph Completion via Pre-Trained Language Models (2023)

Abstract: The mission of open knowledge graph (KG) completion is to draw new findings from known facts. Existing works that augment KG completion require either (1) factual triples to enlarge the graph reasoning space or (2) manually designed prompts to extract knowledge from a pre-trained language model (PLM), exhibiting limited performance and requiring expensive efforts from experts. To this end, we propose TagReal that automatically generates quality query prompts and retrieves support information from large text corpora to probe knowledge from PLM for KG completion. The results show that TagReal achieves state-of-the-art performance on two benchmark datasets. We find that TagReal has superb performance even with limited training data, outperforming existing embedding-based, graph-based, and PLM-based methods.

## introduction a knowledge graph (kg) is a heterogeneous graph that encodes factual information in the form of entity-relation-entity triplets, where a relation connects a head entity and a tail entity (e.g., "miami-located_in-usa") @xcite . kg @xcite plays a central role in many nlp applications, including question answering @xcite , recommender systems @xcite , and drug discovery @xcite . however, existing works @xcite show that most large-scale kgs are incomplete and cannot fully cover the massive real-world knowledge. this challenge motivates kg completion, which aims to find one or more object entities given a subject entity and a relation @xcite . for example, in figure 1 , our goal is to predict the object entity with "detroit" as the subject entity and "contained_by" as the relation. however, existing kg completion approaches @xcite have sevfigure 1 : the quality of hand-crafted prompts can be limited, while prompt mining is a scalable alternative. support information also helps plm understand the purpose of prompts. in this example, canada and michigan are potentially valid options but given prompt mining and support information retrieval, the model becomes confident about michigan as the answer here. pre-trained language models (plms) @xcite have shown to be powerful in capturing factual knowledge implicitly from learning on massive unlabeled texts @xcite . since plms are superb in text encoding, they can be utilized to facilitate knowledge graph completion with external text information. recent knowledge graph completion methods @xcite focus on using manually crafted prompts (e.g., "detroit is located in @xcite " in figure 1 ) to query the plms for graph completion (e.g., "michigan"). however, manually creating prompts can be expensive with limited quality (e.g., plm gives a wrong answer "canada" to the query with a handcrafted prompt, as shown in figure 1 ). building on the above limitations of standard kg and the enormous power of plms @xcite , we aim to use plms for open knowledge graph completion. we propose an end-to-end framework that jointly exploits the implicit knowledge in plms and textual information in the corpus to perform knowledge graph completion (as shown in figure 1 ). unlike existing works (e.g., @xcite ), our method does not require a manually pre-defined set of facts and prompts, which is more general and easier to adapt to real-world applications. our contributions can be summarized as: ## methodology we propose tagreal, a plm-based framework to handle kg completion tasks. in contrast to the previous work, our framework does not rely on handcrafted prompts or pre-defined relevant facts. as shown in figure 2 , we automatically create appropriate prompts and search relevant support information, which are further utilized as templates to explore implicit knowledge from plms. ## conclusion and future works in this study, we proposed a novel framework to exploit the implicit knowledge in plm for open kg completion. experimental results show that our method outperforms existing methods especially when the training data is limited. we showed that the optimized prompts with our approach outperform the handcrafted ones in plm knowledge probing. the effectiveness of the support information retrieval to aid the prompting is also demonstrated. in the future, we may leverage qa model's power to retrieve more reliable support information. another potential extension is to make our model more explainable by exploring path-finding tasks.