SAC - KG : Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graph (2024)

Abstract: Knowledge graphs (KGs) play a pivotal role in knowledge-intensive tasks across specialized domains, where the acquisition of precise and dependable knowledge is crucial. However, existing KG construction methods heavily rely on human intervention to attain qualified KGs, which severely hinders the practical applicability in real-world scenarios. To address this challenge, we propose a general KG construction framework, named **SAC-KG**, to exploit large language models (LLMs) as **S**killed **A**utomatic **C**onstructors for domain **K**nowledge **G**raph. SAC-KG effectively involves LLMs as domain experts to generate specialized and precise multi-level KGs. Specifically, SAC-KG consists of three components: Generator, Verifier, and Pruner. For a given entity, Generator produces its relations and tails from raw domain corpora, to construct a specialized single-level KG. Verifier and Pruner then work together to ensure precision by correcting generation errors and determining whether newly produced tails require further iteration for the next-level KG. Experiments demonstrate that SAC-KG automatically constructs a domain KG at the scale of over one million nodes and achieves a precision of 89.32%, leading to a superior performance with over 20% increase in precision rate compared to existing state-of-the-art methods for the KG construction task.

## introduction knowledge graphs (kgs) are a collection of factual triples, which represent human knowledge in a structured way, i.e., (head entity, relation, tail entity). in recent years, kgs have been successfully applied in various domains, including medical science @xcite , biology @xcite , and social networks @xcite . however, constructing domain kg requires extensive expert knowledge and human intervention, which severely restricts the practical implementation of domain kg construction. to address this challenge, extensive research efforts have been devoted to the kg construction task @xcite . canonical kg construction methods mainly focus on learning logical rules based on semantic patterns. rule-based methods extract subjectpredicate-object triples by utilizing lexical and semantic role labels @xcite . recently, some large language models (llms)-based methods have emerged as a new trend and achieved superior performances than rule-based methods @xcite . llm-based methods extract triples from raw corpora by harnessing the prior knowledge stored within the llm. extensive works demonstrate that llm-based methods are more creative @xcite and more human-understandable @xcite . albeit with multiple benefits of the llm-based methods, they confront two significant challenges that severely hinder their performance and deployment. first, there is contextual noise in input. existing llm-based methods extract triples directly from the raw context. the raw context includes a substantial amount of domain-irrelevant information, which may potentially distract the llm and consequently degrade its performance @xcite . second, there is knowledge hallucination in output. knowledge hallucination is that the llm may generate content that is nonsensical or unfaithful to the provided source content @xcite . regarding the domain kg construction, the llm might generate certain irrelevant or incorrect triples due to the contextual noise and knowledge hallucination. moreover, these incorrect triples may further propagate their errors to the next itera- domain corpora open knowledge [growing] [growing] [growing] [pruned] [growing] [pruned] [pruned] …… [correct] [correct] [correct] [correct] [format error] [correct] [correct] [correct] …… pruner verifier generator llm figure 1 : an example of input and output of the sac-kg framework. specifically, the input component consists of three segments: text, instruction, and examples. the text segment retrieves the most relevant corpora from a domain-specific corpora for a given entity. the instruction segment provides instructions to an llm to generate corresponding triples. the example segment retrieves template triples from an open-source encyclopedia kg. the output includes generated correct triples and an indicator of "growing" or "pruned" by pruner. therefore, in this paper, we seek to answer the question: can we propose a general kg construction framework that is automatic, specialized, and precise? with this consideration, we delve explicitly into the two significant challenges and propose a novel approach, namely sac-kg, which involves the llm as domain experts and constructs domain kgs by an entity-induced tree search algorithm automatically and iteratively. sac-kg is a novel automatic kg construction framework and effectively addresses the issues mentioned above within llm-based methods. specifically, sac-kg comprises three components: in-context learning. in-context learning (icl), where llms make predictions only based on contexts with a few examples, has become a new paradigm for natural language processing @xcite . with the scaling of both model size and training corpora size @xcite , llms demonstrate the ability of learning from a few prompts that contain some training examples @xcite . different from supervised learning requiring a training stage that uses backward gradients to update model parameters, icl does not need parameter update and directly performs predictions. icl aims to learn from analogy, which directly llms to make predictions @xcite with these examples. by concatenating both context and prompt, llms learn patterns hidden in examples and perform well on downstream tasks @xcite . ## method we develop a general framework, named sac-kg, to exploit llms (see appendix a for related works) as skilled automatic constructors for domain kgs. given domain corpora, the overall task is to extract triples with automation, precision, and controllability. sac-kg organically integrates generator, verifier, and pruner in a unified framework to perform kg construction. an overview of sac-kg is shown in figure 2 . ## experiments we design experiments to evaluate the effectiveness of the proposed sac-kg and provide more insights of the constructed domain kg. with this desiderata, we divide the experiments into five parts: to evaluate the effectiveness of sac-kg, we compare sac-kg with existing state-of-theart methods for domain kg construction task. (ii) to offer a more comprehensive evaluation of constructed kg, we conduct agreement evaluation between gpt4 and humans. (iii) to provide more insight into sac-kg, we conduct the ablation study of each component. (iv) to analyze the constructed kg, we conduct case study of the constructed domain kg. (v) to further demonstrate the effectiveness of sac-kg, we evaluate sac-kg on existing ## conclusion in this paper, we propose a novel automatic domain kg construction framework named sac-kg, which effectively constructs kg directly from domain corpora. sac-kg incorporates llms as domain experts and iteratively employs an entityinduced tree search algorithm for the construction of a multi-level kg. specifically, we propose generator, verifier, and pruner to form a general kg construction framework with automation, precision, and controllability. sac-kg constructs a domain kg at the scale of over a million nodes with an precision of 89.32%, achieving over 20% increase in precision metric. this superior performance of sac-kg over existing state-of-the-art methods demonstrates effectiveness of our sac-kg. a more related works language models. language models including gpt @xcite , bert @xcite , roberta @xcite , and megatron-lm @xcite have led to a learning paradigm shift in natural language processing (nlp). models are first pre-trained on extensive volumes of unlabeled text corpora with language modeling objectives, and then fine-tuned on downstream tasks. recently, large language models (llms) including chatgpt (openai, 2020) and palm @xcite have shown great performance in both few-shot and even zero-shot scenarios @xcite . to further enhance the interpretability of these llms, some research endeavors explain llms through attribution analysis @xcite @xcite . another line of work aims to retrieve the knowledge explicitly from llms as the basis for interpreting them, including the reasoning task @xcite and the qa task @xcite @xcite .