Extract, Define, Canonicalize: An LLM -based Framework for Knowledge Graph Construction (2024)

Abstract: In this work, we are interested in automated methods for knowledge graph creation (KGC) from input text. Progress on large language models (LLMs) has prompted a series of recent works applying them to KGC, e.g., via zero/few-shot prompting. Despite successes on small domain-specific datasets, these models face difficulties scaling up to text common in many real-world applications. A principal issue is that, in prior methods, the KG schema has to be included in the LLM prompt to generate valid triplets; larger and more complex schemas easily exceed the LLMs’ context window length. Furthermore, there are scenarios where a fixed pre-defined schema is not available and we would like the method to construct a high-quality KG with a succinct self-generated schema. To address these problems, we propose a three-phase framework named Extract-Define-Canonicalize (EDC): open information extraction followed by schema definition and post-hoc canonicalization. EDC is flexible in that it can be applied to settings where a pre-defined target schema is available and when it is not; in the latter case, it constructs a schema automatically and applies self-canonicalization. To further improve performance, we introduce a trained component that retrieves schema elements relevant to the input text; this improves the LLMs’ extraction performance in a retrieval-augmented generation-like manner. We demonstrate on three KGC benchmarks that EDC is able to extract high-quality triplets without any parameter tuning and with significantly larger schemas compared to prior works. Code for EDC is available at https://github.com/clear-nus/edc.

## introduction knowledge graphs (kgs) @xcite are a structured representation of knowledge that organizes interconnected information through graph structures, where entities and relations are represented as nodes and edges. they are broadly used in a variety of downstream tasks such as decision-making @xcite , question-answering @xcite , and recommendation @xcite . however, knowledge graph construction (kgc) is inherently challenging: the task requires competence in understanding syntax and semantics to generate a consistent, concise, and meaningful knowledge graph. as such, kgc predominantly relies on intensive human labor @xcite . kgc is a broad problem and in this work, we focus on the task of relational triplet extraction as it is crucial for kgc. following previous works @xcite @xcite , we still refer to the task we are addressing as kgc. recent attempts to automate kgc @xcite have employed large language models (llms) in view of their remarkable natural language understanding and generation capabilities. llm-based kgc methods employ various innovative prompt-based techniques, such as multi-turn conversation @xcite and code generation @xcite , to generate entityrelation triplets that represent the knowledge graph. however, these methods are currently limited to small and domain-specific scenarios -to ensure the validity of generated triplets, schema information (e.g., possible entity and relation types) has to be included in the prompt. complex datasets (e.g., wikipedia) typically require large schemas that exceed the context window length or can be ignored by the llms @xcite . furthermore, pre-defined schemas are not always available -the users might not have pre-determined or fixed intentions about what information is of interest in advance but still would like to extract intrinsically high-quality kgs. it is unclear how existing methods will work in such situations. to address these problems, we propose extract-define-canonicalize (edc), a structured approach for kgc: the key idea is to decompose kgc into three primary phases corresponding to three subtasks (fig. 1 ): each phase exploits the strengths of llms: the extract subtask leverages recent findings that llms are effective open information extractors @xcite -they can extract semantically correct and meaningful triplets. however, the resulting triplets typically contain redundant and ambiguous information, e.g., multiple semantically equivalent relation phrases such as 'profession', 'job', and 'occupation' @xcite @xcite . phases 2 and 3 (define and canonicalize) standardize the triplets to make them useful for downstream tasks. we designed edc to be flexible: it can either discover triplets consistent with a preexisting schema of potentially large size (target alignment) or self-generate a schema (self canonicalization). to achieve this, we use llms to define the schema components by exploiting their explanation generation capabilities -llms can justify their extractions via explanations that are agreeable to human experts @xcite . the definitions are used to find the closest entity/relation type candidates (via a vector similarity search) that the llm can then reference to canonicalize a component. in the case there is no equivalent counterpart in the existing schema, we can choose to add it to enrich the schema. to further improve performance, the three steps above can be followed by an additional refinement phase: we repeat edc but provide the previously extracted triplets and a relevant part of the schema in the prompt during the initial extraction. we propose a trained schema retriever that retrieves schema components relevant to the input text, akin to retrieval-augmented generation @xcite , which we find improves the generated triplets. experiments on three kgc datasets in both target alignment and self canonicalization settings show that edc is able to extract higher-quality kgs compared to state-of-the-art methods through both automatic and manual evaluation. furthermore, the use of the schema retriever is shown to significantly and consistently improve edc's performance. in summary, the paper makes the following contributions: ## background in this section, we provide relevant background on knowledge graph construction (kgc), open information extraction (oie), and canonicalization. ## method: edc for kgc this section outlines our primary contribution: an approach to constructing knowledge graphs that leverages llms in a structured manner. we first detail the edc framework followed by a description of refinement (edc+r). given input text, our goal is to extract relational triplets in a canonical form such that the resulting kgs will have minimal ambiguity and redundancy. when there is a predefined target schema, all generated triplets should conform to it. in the scenario where there is not one, the system should dynamically create one and canonicalize the triplets with respect to it. ## experiments in this section, we describe experiments designed to evaluate the performance of edc and edc+r. briefly, our results demonstrate that edc significantly outperforms the state-of-the-art methods in both target alignment and self canonicalization settings. refinement further improves edc. source code for edc and to replicate our experiments are available in the supplementary materials, with full tables in the appendix c. ## conclusion in this work, we presented edc, an llm-based three-phase framework that addresses the problem of kgc by open information extraction followed by post-hoc canonicalization. experiments show that edc and edc+r are able to extract better kgs than specialized trained models when a target schema is available and dynamically create a schema when none is provided. the scalability and versatility of edc opens up many opportunities for applications: it allows us to automatically extract high-quality kgs from general text using large schemas like wikidata @xcite and even enrich these schemas with newly discovered relations.