B ert N et: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models (2023)

Abstract: It is crucial to automatically construct knowledge graphs (KGs) of diverse new relations to support knowledge discovery and broad applications. Previous KG construction methods, based on either crowdsourcing or text mining, are often limited to a small predefined set of relations due to manual cost or restrictions in text corpus. Recent research proposed to use pretrained language models (LMs) as implicit knowledge bases that accept knowledge queries with prompts. Yet, the implicit knowledge lacks many desirable properties of a full-scale symbolic KG, such as easy access, navigation, editing, and quality assurance. In this paper, we propose a new approach of harvesting massive KGs of arbitrary relations from pretrained LMs. With minimal input of a relation definition (a prompt and a few shot of example entity pairs), the approach efficiently searches in the vast entity pair space to extract diverse accurate knowledge of the desired relation. We develop an effective search-and-rescore mechanism for improved efficiency and accuracy. We deploy the approach to harvest KGs of over 400 new relations, from LMs of varying capacities such as RoBERTaNet. Extensive human and automatic evaluations show our approach manages to extract diverse accurate knowledge, including tuples of complex relations (e.g., “A is capable of but not good at B”). The resulting KGs as a symbolic interpretation of the source LMs also reveal new insights into the LMs’ knowledge capacities.

## introduction symbolic knowledge graphs (kgs) are a powerful tool for indexing rich knowledge about entities and their relationships, and are useful for information access @xcite , decision making @xcite , and improving machine learning in general @xcite @xcite . ## related work knowledge graph construction popular knowledge bases or kgs are usually constructed with heavy human labor. for example, wordnet @xcite is a lexical database that links words into semantic relations; conceptnet @xcite is a large commonsense knowledge graph presented as a set of knowledge triples; atomic @xcite is a crowd-sourced social commonsense kg of if-then statements. recently, automatic knowledge base construction (akbc) as a research focus has led to various approaches (summarized in lms as knowledge bases another line of works attempted to use lms as knowledge bases (lama, @xcite . these works are also known as factual probing because they measured how much knowledge is encoded in lms. this is usually implemented by prompting methods and leveraging the masked lm pretraining task. lpaqa @xcite proposes to use text mining and paraphrasing to find and select prompts to optimize the prediction of a single or a few correct tail entities, instead of extensively predicting all the valid entity pairs like in our framework. autoprompt @xcite , @xcite @xcite learn discrete or continuous prompts automatically with an additional training set. though making prompts unreadable, these methods achieve higher accuracy on the knowledge probing tasks. our framework differs from these works in that we aim to explicitly harvest knowledge graphs instead of measuring the knowledge in a simplified setting. consistency of lms consistency is a significant challenge for lms, which stresses that they should not produce conflicting predictions across inference sessions. for example, models should be-have invariantly under inputs with different surface forms but the same meaning. @xcite analyzed the consistency of pretrained lms with respect to factual knowledge. jiang et al. 2020 used paraphrasing to improve factual probing. newman et al. 2021 trains an additional layer on top of word embedding to improve consistency. recently, consistency is also shown helpful to improve the reasoning ability of large lms @xcite @xcite . in our framework, the extracted entity pairs for each relation are enforced to consistently satisfy a diverse set of prompts and regularized by several scoring terms. ## harvesting kgs from lms this section presents the proposed framework for extracting a relational kg from a given pretrained lm, where the lm can be arbitrary fillin-the-blank models such as bert @xcite , roberta @xcite , bart @xcite , or gpt-3 (with appropriate instructions) @xcite . the kg consists of a set of knowledge tuples in the form ⟨head en-tity (h), relation (r), tail entity (t)⟩. our approach utilizes the lm to automatically harvest a large number of appropriate entity pairs (h 1 , t 1 ), (h 2 , t 2 ), . . ., for every given relation r. this presents a more challenging problem than traditional lm probing tasks, which typically predict a single tail entity or a small number of valid tail entities given a head entity and relation. our approach for extracting knowledge tuples of a specific relation of interest, such as "potential_risk" as depicted in figure 2 , only requires minimal input information that defines the relation. this includes an initial prompt, such as "the potential risk of a is b" and a small number of example entity pairs, such as ⟨eating candy, tooth decay⟩. the prompt provides the overall semantics of the relation, while the example entity pairs clarify possible ambiguities. for new relations not included in existing kgs, it is impractical to require a large set (e.g., hundreds) of example entity pairs as in previous knowledge probing or prompt optimization methods @xcite @xcite . in contrast, our approach necessitates only a small number of example entity pairs, for example, as few as 2 in our experiments, which can easily be collected or written by users. ## experiments to evaluate our framework, we extract knowledge of diverse new relations from various language models, and conduct human evaluation. we then make deeper analysis of prompt creation and scoring function in our framework. finally, by utilizing our framework as a tool to interpret the knowledge stored in language models, we have made noteworthy observations regarding the knowledge capacity of black-box models. ## conclusion we have developed an automatic framework that extracts a kg from a pretrained lm (e.g, bert, roberta), in an efficient and scalable way, resulting in a family of new kgs, which we refer to as bertnet, robertanet, etc. our framework is capable of extracting knowledge of arbitrary new relation types and entities, without being restricted by pre-existing knowledge or corpora. the resulting kgs also serve as interpretation of source lms. limitations our current design and experimental studies are limited on lms in the generic domain, and are not yet been studied in specific domains such as extracting healthcare knowledge from relevant neural models. we leave the exciting work of harvesting knowledge from various kinds of neural networks across applications and domains in the future work.