Can LLM s be Knowledge Graph Curators for Validating Triple Insertions? (2025)

Abstract: As Knowledge Graphs (KGs) become central to modern applications, automated methods for validating RDF triples before insertion into these graphs are essential. The complexity and scalability challenges in manual validation processes have led researchers to explore Large Language Models (LLMs) as potential automated validators. This study investigates the feasibility of using LLMs to validate RDF triples by focusing on four distinct and complementary validation tasks: class and property alignment, URI standardization, semantic consistency, and syntactic correctness. We propose a systematic validation method that uses prompts to guide LLMs through each stage of the triple evaluation of the RDF. In our experiments, four models are evaluated across these tasks. Our results reveal that more advanced models like Llama-3-70B-Instruct offer superior accuracy and consistency. Our findings emphasize the practical open challenges of deploying LLMs in real-world RDF validation scenarios, including domain generalization, semantic drift, and the need for human-in-the-loop interventions. This investigation advances the research on the refinement and integration of LLM-based RDF validation techniques into KG management workflows.

## introduction knowledge graphs (kgs) have emerged as essential artifacts to represent structured knowledge in various digital applications, such as search engines, recommendation systems, and question-answering platforms. having underlying logical and semantically consistent kgs is relevant for applications because they rely on them to improve user experience and help decision making. at the core of kgs are resource description framework (rdf) triples, which consist of subjectpredicate-object expressions that form the basic building blocks of kgs. an rdf triple links a subject to an object through a predicate, encapsulating a single piece of knowledge @xcite . in this context, ontologies play a significant role in kgs by defining structured schema by ensuring consistency and semantic integrity within kgs. ontologies specify the classes, properties, and relationships that form the backbone of kgs by guiding the data management and querying processes. maintaining the integrity and consistency of kgs as new rdf triples are added is a complex and ongoing challenge. traditional methods for validating and inserting rdf triples often involve manual efforts by ontology experts, which can be time consuming and prone to human errors. these methods struggle to keep pace with modern data environments' dynamic and large-scale nature. the limitations of current approaches highlight the urgent need for advanced automated tools that can support ontology experts in the management of kg. automating the identification and elimination of erroneous information improves efficiency and accuracy, reducing the dependency on extensive human intervention. the insertion of new rdf triples into an existing kg presents issues that can undermine the graph's reliability and usability. first, there is the problem of violating the predefined classes and properties in place, where new triples might not conform to the established ontology schema. second, uri standardization and duplication pose significant challenges; ensuring that new triples do not introduce redundant or conflicting uris is essential for maintaining a coherent kg. third, semantic inconsistency is an issue, as newly added triples might contradict existing knowledge, leading to logical inconsistencies within the graph. lastly, the syntactic correctness of the triples, respecting a pre-defined language (e.g., n-triples @xcite and turtle @xcite syntax), avoids malformed triple errors in rdf parsers. these issues collectively impact the overall effectiveness of kgs, compromising their ability to deliver accurate and reliable knowledge representation. recent advancements in language models (llms) have opened new avenues for addressing the challenges associated with kg management. llms, which excel in various natural language processing (nlp) tasks, have demonstrated capabilities in understanding and generating human language with contextual and semantic accuracy @xcite . the intersection of kgs and llms presents a promising opportunity to leverage these models to enhance kg management processes @xcite . we originally hypothesized that the advanced semantic understanding of llms could assist in identifying violations of classes and properties, standardizing uris, and ensuring syntactic and semantic consistency of triples. this integration has the potential to significantly improve the efficiency and reliability of kg management, providing ontology experts with powerful tools to maintain high-quality kgs. this study investigates and evaluates the use of llms in validating and inserting rdf triples into existing kgs without negatively impacting their integrity. specifically, we develop a methodology that ensures new triples are consistent with the existing kg and conform to underlying ontologies. the broader implications of this research include potential benefits for both academia and industry, such as more reliable kgs and improved data management processes over time. integrating llms into kg curation tasks can lead to more intelligent and automated knowledge management systems, offering enhanced capabilities for handling complex and dynamic data environments. this article is organized as follows: section 2 reviews related work, discussing previous approaches for rdf triple validation. section 3 defines the addressed issues in validating rdf triples. section 4 outlines our designed method. section 5 presents the evaluation procedures and obtained results. section 6 discusses our findings and open research challenges. finally, section 7 draws conclusion remarks. ## related work this section summarizes existing investigations and approaches to validating rdf triples. a recently published survey describing the intersection between llms and kgs @xcite identified kg validation as an essential research venue. kg validation is categorized into two main approaches: fact-checking and inconsistency detection. our present solution concentrates on inconsistency detection, a relatively underexplored area within the broader context of kg validation. the survey highlights only one significant study in this domain: chatrule @xcite . cha-trule is a framework that leverages kgs to build llm prompts, generating rules to detect inconsistencies within the kg. our work further extends this field by systematically evaluating the capability of llms to validate rdf triples in kg insertion operations, focusing on various types of inconsistencies. huaman and fensel presents a methodical approach to improving kg quality without using llms @xcite . the framework integrates existing tools and workflows to ensure correctness, completeness, and usability of kgs. it employs rule-based methods for quality assessment, using metrics like accuracy and completeness. cleaning tasks involve schema verification through constraint languages (e.g., shacl, shex) and fact validation using internal consistency checks or external sources like wikipedia. for enrichment, it detects duplicates and resolves conflicts using tools such as silk and limes. frey et al. @xcite demonstrated the evaluation of various llms, including gpt-4 foot_0 and claude 2 foot_1 , revealing their proficiency in working with turtle, an rdf triple serialization format. their study introduced some tasks to assess the models' ability to parse, understand, and create kgs in turtle syntax. while newer versions of gpt and claude demonstrate promising capabilities, they frequently struggle with strict output formatting, often including unnecessary explanations, complicating their integration with rdf tools. we face similar problems with our method and despite these challenges, the models show huge potential for assisting in kg engineering. the triples accuracy assessment (taa) @xcite approach offers an automated method for validating rdf triples in a kg using other kgs. unlike traditional methods that rely on internal information, taa identifies equivalent resources across different kgs and matches predicates to assess the correctness of triples. a confidence score is generated to indicate the accuracy of each triple, showing promising results with high f-measure scores in evaluations using the factbench dataset. our originality lies in the innovative use of llms to validate rdf triples for kg insertion operations, which traditionally solely rely on rulebased methods or external kg interlinks. unlike prior approaches, such as the triples accuracy assessment @xcite , which leverages other kgs for validation, our study explores the potential of llms to bring deeper semantic understanding and context to the validation process. to the best of our knowledge, this study is the first to systematically assess the potential effectiveness of llms in this specific application and across various rdf validation tasks. our study offers new, original insights into llms' potential to enhance the accuracy and efficiency of rdf triple validation. ## problem formulation this section outlines four critical problems our approach addresses when validating and inserting new triples into a kg using llms. the rationale behind choosing the following problems is that they align with existing standards and best practices in rdf and ontology management. they are common underlying problems encountered during the construction and maintenance of kgs. ## validating generated rdf triples based on llms our proposed method consists of four main steps, each involving a specific prompt and requiring the intervention of an ontology maintainer to ensure correctness. these steps systematically validate and prepare rdf triples for precise insertion into an existing kg. the steps address the critical issues of class and property compliance, uri uniqueness, semantic and syntactic consistency, and challenges explained with more details in section 3. figure 1 presents our method to validate rdf triples. the input is a set of rdf triples formatted as algorithm 1 shows the procedure of our method. the first step (#1 in figure 1 ) aims to verify that t contains classes and properties listed as necessary by the ontology maintainer. the process begins with the maintainer creating a list of important classes and properties l c,p (line 2 in algorithm 1). this list outlines the crucial classes and properties that must be present in the rdf triples. the list l c,p is manually curated by the ontology maintainer, who possesses a deep knowledge of the kg's structure and the relevant domain. this list is derived directly from the ontology (cf. figure 1 ). although the important classes and properties human-curated lists may limit generalizability, they are important for ensuring semantic coherence and alignment with the kgs domain. these curated inputs are minimal compared to the automated processing enabled by llms in the pipeline. the llm evaluates each triple from all triples t (line 3 of algorithm 1) to check for compliance with the provided list. the compliance check is defined as: ∀(s, p, o) ∈ t , (class(s) ∈ l c,p ) ∧ (property(p) ∈ l c,p ). any triples containing classes or properties not included in the list of important classes and properties are flagged (lines 5 and 6 of algorithm 1). the ontology maintainer reviews these noncompliant triples and determines whether they should be removed (line 9 of algorithm 1). this step ensures that all generated triples adhere to the predefined schema. the second step (#2 in figure 1 ) ensures that new triples do not introduce duplicate resources into the kg. after removing the triples flagged in the ontology maintainer reviews the flagged duplicates and updates the triples as necessary. if a resource is confirmed as duplicate, the maintainer updates the triples to use the correct, existing resource values (line 18 of algorithm 1). if a resource is erroneously marked as a duplicate, it is ignored. this step guarantees the uniqueness of uris in the kg, preventing conflicts and ensuring a coherent representation of entities. the third step (#3 in figure 1 ) ensures that the new triples do not violate predefined semantic restrictions. the ontology maintainer provides a list of semantic restrictions l sr (line 21 of algorithm 1), primarily consisting of rules specified in swrl. the set of triples modified by the previous steps t is then compared against these restrictions by the llm (line 22 of algorithm 1). the language model identifies any triples that potentially violate the semantic rules. the prompt for this step pr 3 = (i 3 , t , l sr ) includes an initial instruction i 3 on identifying triples with semantic restrictions, the set of rdf triples to be analyzed t and the list of semantic restrictions l sr . the ontology maintainer reviews the flagged triples and decides whether to remove them (line 27 of algorithm 1). this step prevents the introduction of logical contradictions, such as an object being simultaneously marked as compatible and incompatible with another object, thus maintaining the semantic integrity of the kg. the final step (#4 in figure 1 ) ensures the syntactic correctness of the rdf triples before they are inserted into the kg. the set of triples modified by the previous steps, t , is provided as input, and the language model checks for any syntactic errors (line 30 of algorithm 1). this step does not require additional lists as the previous steps. the language model identifies and flags triples that do not conform to the required rdf structure (line 34 of algorithm 1), ensuring that only syntactically correct triples are considered for insertion into the kg. the prompt for this step pr 4 = (i 4 , t ) includes an initial instruction i 4 about the syntactic validation and the set of rdf triples to be analyzed t . the ontology maintainer proceeds with a final validation on the flagged triples and t f inal , ensuring they are ready to be inserted in the kg. ## evaluation this evaluation assesses if the developed method and the designed prompts instructing the llms can effectively identify and correct specific issues within the rdf triples. the evaluation focuses on the four distinct problems identified in section 3 and addressed by our solution (section 4). section 5.1 describes the models, datasets, and procedures used in this evaluation. section 5.2 demonstrates the obtained results. ## discussion and open research challenges this research inquired how llms can be suited to contribute as kg curators in the operations of triple insertion. this research demonstrated that llms for the distinct problems addressed can be applicable as an approach to help ontology engineers address rdf validation. in the following, we underline key findings and challenges regarding several aspects of our experimental results and the consequences of applying our solution to operational settings. the most performing llm. overall, we found that the llama-3 70b instruct model consistently outperformed the others across all validation problems, excelling in tasks that demand high precision and recall. its effectiveness in the "syntactic inconsistency problem" (0.99 accuracy) and "uri standardization problem" (0.96 accuracy) underscored its robustness in handling structural data such as rdf triples. in our understanding, this model's success is due to its large parameter size and fine-tuning, which are geared explicitly towards instruction-based tasks, enabling it to generalize across diverse rdf validation scenarios. underperformance consistently. conversely, bloom-176b consistently underperformed, particularly in the "semantic inconsistency problem" (0.29 accuracy) and "syntactic inconsistency problem" (0.36 accuracy). its lower accuracy and inconsistent precision-recall balance show its limitations in handling the rule-based nature of rdf validation. the gap in results between llama-3 and bloom can be explained by differences in model size, training datasets, domain-specific tuning, and more than two years between the release of both models. the most challenging problems. discussing the four validation problems, the "uri standardization problem" obtained the best overall results, with a mean accuracy of 0.71. this can be attributed to the nature of 'standardizing uris', which primarily involves pattern recognition that llms are well-equipped to handle. the best model, llama-3, achieved a near-perfect accuracy of 0.96 for this problem, showing its ability to manage standardized data consistently. on the contrary, the "syntactic inconsistency problem" proved to be the most challenging overall, with an average accuracy of 0.53 and a mean f1 score of 0.47. this difficulty arose from gemma 2 reaching a precision of 0.08 and accuracy of 0.26, which was the worst precision and accuracy of the evaluation. mixtral-7b got better results in this task. comparing the two models with similar sizes, mixtral outperformed gemma 2 in semantic-related tasks, and gemma 2 outperformed mixtral in syntactic-related tasks. cost vs. accuracy trade-off. the costeffectiveness of deploying different llms for rdf triple validation is critical for real-world applications. for instance, while the llama model achieved superior accuracy and overall metrics, it comes with a significant computational cost of @xmath00.30 per million tokens, provides a balanced trade-off between cost and accuracy but falls short of achieving the precision needed for more complex scenarios. mixtral 7b instruct offers a middle ground in cost and performance at $0.60 per million tokens. at the same time, bloom is a freely available model that, despite being cost-free, exhibits significantly lower accuracy and reliability. these costs were gathered in two companies that provide llms apis: to-getherai foot_7 and hugging face foot_8 . the costs reflect the price found when this manuscript was written -september 2024. semantic drift in long triple chains. one issue encountered in rdf triple validation using llms is the potential for semantic drift when evaluating long chains of interconnected triples. in this context, semantic drift refers to the model losing coherence as it processes extended sequences. this drift is increased by the models' limited memory retention and inability to consistently track relationships across multiple triples. triples involving big and deep ontological hierarchies or chains that span various levels may introduce errors as the models struggle to maintain context. as a future work, addressing this challenge may require fine-tuning llms with specific datasets designed to enhance memory retention over long sequences or integrating mechanisms that allow for continuous context tracking in kg context. without such interventions, long triple chains remain a source of inaccuracy. ontology complexity and coverage. the complexity of ontologies, characterized by rich hierarchies, specialized vocabularies, and relationships, introduces significant challenges for llm-based rdf validation. the method revealed that as ontologies grow more complex, models like bloom and mixtral struggle to navigate the intricate set of classes and properties accurately. a notable issue is incomplete ontology coverage, where the models lack sufficient information about specialized vocabularies, leading to false positives or negatives during validation. for example, triples involving lesser-known properties or deep subclass hierarchies often went unrecognized, highlighting gaps in the models' ontological understanding. addressing this issue may require expanding the training datasets to include more comprehensive ontology samples for future work. ## conclusion ensuring the quality and consistency of kgs is critical for real-world applications that rely on semantic accuracy. as kgs become more integral to artificial intelligence systems, advancing methods for their automated validation might play a key role in driving accurate, reliable, and scalable semantic solutions. our study explored using llms to validate rdf triples by addressing critical challenges in automating a traditionally manual process. we showcased the strengths and limitations of current llms in kg curation by examining the effectiveness of models like llama-3-70b-instruct and bloom-176b across four rdf validation tasks. the llama-3 model demonstrated competitive results, particularly in maintaining syntactic and semantic consistency, showing the potential for real-world deployment. our results highlighted the complexity and cost implications, especially in handling errors requiring more context. the findings suggested future research directions, including more sophisticated approaches to reducing semantic drift in longer triple chains and enhancing model generalization across domains. also, incorporating humans into the loop and refining prompt engineering techniques could enhance llm results.