Generating Domain-Specific Knowledge Graphs from Large Language Models (2025)

Abstract: Knowledge graphs (KGs) have been a cornerstone of search and recommendation due to their ability to store factual knowledge about any domain in a structured form enabling easy search and retrieval. Large language models (LLMs) have shown impressive world knowledge across different benchmarks and domains but their knowledge is inconveniently scattered across their billions of parameters. In this paper, we propose a prompt-based method to construct domain-specific KGs by extracting knowledge solely from LLMs’ parameters. First, we use an LLM to create a schema for a specific domain, which contains a set of domain-representative entities and relations. After that, we use the schema to guide the LLM through an iterative data generation process equipped with Chain-of-Verification (CoVe) for increased data quality. Using this method, we construct KGs for two domains: books and landmarks, which we then evaluate against Wikidata, an open-source human-created KG. Our results show that LLMs can generate large domain-specific KGs containing tens of thousands of entities and relations. However, due to the increased hallucination rates as the procedure evolves, the utility of large-scale LLM-generated KGs in practical applications could remain limited.

## introduction large language models (llms) have emerged as a centerpiece of nlp. these models learn to understand and produce natural language(s) while also encoding a vast amount of world knowledge from the terabytes of text they "see" during their pre-training @xcite @xcite . even early variants of pre-trained language models were found to contain rich factual knowledge @xcite . these models @xcite contained millions of parameters and were trained on gigabytes of text, while the latest llms contain billions of parameters and are trained on terabytes of text containing resources such as scientific papers, wikipedia, books, and web. as a consequence, they have been found to store a remarkable knowledge about many diverse domains @xcite @xcite and their abilities are commonly evaluated on open-domain question answering benchmarks such as triviaqa @xcite , naturalquestions @xcite and openbookqa @xcite . the knowledge stored in llms is a by-product of their training procedure. however, this knowledge is encoded within their parameters which makes it difficult and inefficient to extract or use. knowledge graphs (kgs) store large-scale factual knowledge in a structured form, which enables an easy integration with query languages and search engines @xcite . they serve knowledge from different domains in the form of entities (nodes) and relations (edges) which form triples @xcite . however, their construction and maintenance requires human efforts and large costs because kgs need to store comprehensive, up-to-date and correct knowledge @xcite . consequently, methods for kg construction which automatically extract structured knowledge (in the form of triples) from unstructured texts have been developed @xcite @xcite . in addition to generaldomain kgs, such methods also target domainspecific kg construction using corpora such as medical records @xcite or maintenance texts @xcite . however, the applicability of these methods requires access to domain-specific documents which can be difficult or expensive to obtain for all domains of interest. to overcome this constraint and owing to the growing llm memorisation and generation capa-bilities, @xcite proposed a prompt-based procedure to construct a general-domain kg from scratch relying solely on the knowledge encoded in llms parameters. this approach enables creating a large kg (millions of entities and relations), but their procedure does not offer any control over the obtained entity or relation types. this could be undesirable for targeted uses cases or applications where domain or schema are typically fixed @xcite . @xcite proposed to construct a knowledge base centered around a given "seed" entity, but they perform only up to two hops resulting in high-precision kgs with only several dozen facts. like @xcite , they do not have a fixed set of entities and relation types, which offers little control over the generation process and makes the evaluation more challenging. in this paper, we study the feasibility of constructing large, domain-specific kgs from scratch using solely llms. to this end, we propose a prompt-based method for extracting domainspecific knowledge that model has seen during pre-training. more precisely, given a domain of interest, we first generate a set of entities and relations (a schema) which contains relevant properties describing the domain. these entities and relations are then used during an iterative, prompt-based data generation procedure to guide the model towards producing an "exhaustive" kg about the given domain. finally, the quality of the constructed kg is evaluated against ground-truth human-crafted kgs. our method provides a way of representing the llm knowledge about any domain in a structured form. it does so by offering a schemaguided procedure which converts the knowledge stored within llms' billions of parameters into a domain-specific graph with arbitrary number of hops, which can serve different applications. in summary, our main contributions are: 1) we formulate an iterative, prompt-based algorithm for extracting domain-specific kgs directly from llms (llm is used only for inference); 2) we propose to systematically evaluate the quality of obtained kgs against existing human-created kgs; 3) our method goes beyond kg construction by providing insights into domain knowledge stored within llms, and can serve for evaluation or comparison between llms. our code and kgs are publicly available at http://github.com/parovicm/ llm-kg-gen . ## conclusion kgs have gained popularity in different applications due to their ability to store factual knowledge in a structured format, enabling efficient and reliable information search and retrieval. llms have achieved remarkable performance on many benchmarks recently and have been shown to store rich factual knowledge. however, this knowledge cannot be efficiently or conveniently used because it is encoded with their billions of parameters. in this paper, we proposed a procedure to create domainspecific kgs by extracting knowledge from llms' parameters, requiring only a domain name as input. our procedure includes schema generation, data generation and verification. in this way, we generated kgs about books and landmarks containing tens of thousands of entities and relations and evaluated their quality against wikidata. we also showed that llama significantly outperforms qwen on this task, producing higher-precision kgs. our method goes beyond kg construction and can be used for evaluating or comparing different llms.