Ontology-guided Knowledge Graph Construction from Maintenance Short Texts (2024)

Abstract: Large-scale knowledge graph construction remains infeasible since it requires significant human-expert involvement. Further complications arise when building graphs from domain-specific data due to their unique vocabularies and associated contexts. In this work, we demonstrate the ability of open-source large language models (LLMs), such as Llama-2 and Llama-3, to extract facts from domain-specific Maintenance Short Texts (MSTs). We employ an approach which combines ontology-guided triplet extraction and in-context learning. By using only 20 semantically similar examples with the Llama-3-70B-Instruct model, we achieve performance comparable to previous methods that relied on fine-tuning techniques like SpERT and REBEL. This indicates that domain-specific fact extraction can be accomplished through inference alone, requiring minimal labeled data. This opens up possibilities for effective and efficient semi-automated knowledge graph construction for domain-specific data.

## introduction knowledge graphs (kgs) have emerged as a powerful tool for representing complex relationships between entities across various domains and in aiding in various tasks (e.g., in search, recommendation systems, and others) @xcite . constructing a kg presents several challenges. the process requires extracting structured information from unstructured data, such as text, using information extraction (ie) techniques. much research has focused on large, publicly available general-purpose kgs like dbpedia, yago, or wikidata, as well as on domain-specific kgs in areas like medicine @xcite or railway safety @xcite . more recent studies have explored the use of kgs to support industrial maintenance activities @xcite . however, building a maintenance kg involves overcoming several additional obstacles: off-the-shelf natural language processing solutions often fail to handle domain-specific data adequately, existing benchmarks do not align with industrial realities, the costs of annotating domainspecific data can be prohibitive, and the typically low volume of domain-specific data makes it challenging to train robust models that generalize well to new instances @xcite . additional difficulties arise when data evolves (e.g., triggering changes in the label space) necessitating computationally-expensive retraining or fine-tuning of models in traditional approaches. in-context-learning @xcite ) and ontology-guided kg construction from text2kgbench @xcite offer the ability to overcome some of these challenges. both these methods are dynamic and adaptable to changes in the ontology or label space without the need for re-training. in-context learning does not require large collection of annotated labeled data upfront but only at time of inference. ontology-guided kg construction allows for seamless changes to the ontology if desired. this makes these methods particularly useful in domains where ontologies evolve over time. recently, large language models (llms) have demonstrated remarkable capabilities in the ability to perform information extraction @xcite . however, most of this work focuses on general domain datasets, e.g. ace datasets 12 , conll2003 @xcite or ta-cred @xcite and little work exists on specialized domain-specific datasets. an annotated dataset of fine-grained schema and corpora for information extraction of maintenance short texts (mst) recently became publicly available: maintie @xcite . in this work, we show how llms can assist with the knowledge graph construction on domainspecific texts. our contributions are as follows: our work implies that llms are well-suited for building domain-specific knowledge graphs, even with limited supervised data. in addition, if largescale data annotation is required, llms can be combined with a human-in-the-loop process that pre-annotates data at an incremental rate. our code, prompts and data are publicly available foot_3 . ## task description in this work, we consider the task of llm-assisted kg construction as automatically extracting graph structured information (subject, object and (directional) relation) from unstructured text data. in line with text2kgbench, we also regard this task as "given an ontology and text corpora, the goal is to construct prompts to instruct the model to extract facts relevant to the ontology". an example of how this is setup in the prompt is given in figure 1 . ## discussion firstly, we will draw a comparison to the results of maintie @xcite . since we only focus on triplet extraction without entity recognition, a comparison must be done between our work and maintie's evaluation of rebel on loose relation extraction (as it only requires agreement on the relation type and entity spans) @xcite . in a supervised fine-tuning setting called curriculum learning, maintie @xcite obtained an f1-score of 0.77. for comparison, llama-3-70b-instruct matches this score by using 150 examples and obtains 0.69 f1-score using only 20 semantically similar in-context examples, making the performance remarkably close. the effectiveness of using only a few semantically similar examples can significantly improve the model's ability to recognize patterns in the data. however, this performance is only close when comparing it to the largest state-of-the-art open and instruction-tuned models. for llama-3-70b, its low performance is explained that a significant portion of its "generations" are empty, which means its low performance is caused by the model's failure to even generate a sequence with triples at all. the associated performance in terms of hallucination is therefore void, given that empty generations by default don't contain tokens that fall outside the given sentence of the ontology. llama-3-70b without instruction-tuning is thus incapable of performing fact extraction, while instruction-tuning llama-3-8b slightly decreases performance rather than improve it. second, we would like to draw a comparison methodologically between rebel, spert and llms and review differences and corresponding consequences. both rebel and spert use a fine-tuning approach that requires the labelled data to be available upfront. for spert, a relation classifier is used and as it constrains its output to a label space, it doesn't suffer from hallucinations. llms do not require this labelled data for fine-tuning, and, in this work, we have shown that even with few examples they can already be effective. however, this at-inference requirement of llm comes with the drawback of hallucinations and is a subject of research @xcite . lastly, we would like to discuss the implications of our findings. building domain-specific knowledge graphs is a time-consuming effort, and building nlp-pipelines to do this often requires considerable resources. our work implies that an incremental human-in-the-loop process could significantly assist with fact extraction. in @xcite , pre-annotation was done by fully fine-tuning spert on an already annotated corpus and annotating a second corpus. our work im-plies that by using llms and in-context learning, pre-annotation could start both earlier (using few examples) and continuously (building the number of examples as you go) using inference-only. this could considerably reduce workload for domain experts that need to be involved. ## conclusion this study explores the use of large language models for constructing knowledge graphs from maintenance short texts. we assess models from the llama family, focusing on fact extraction through two main methods: 1) ontology-guided triplet extraction and 2) in-context learning. utilizing these techniques with the llama-3-70b-instruct model, we achieve fact extraction performance comparable to the current state-of-the-art methods that require fine-tuning. during this process, the issue of hallucinations (incorrect or fabricated information) can arise, often exacerbated by suboptimal settings for token prediction penalties. however, for the llama-3-70b-instruct model, hallucinations are almost non-existent. for other models, it's feasible to prune hallucinated triples from the output. this capability extends even to smaller models like llama-3-8b, making them viable alternatives. this approach facilitates human-in-the-loop preannotation for domain-specific datasets, potentially reducing the time investment required from domain experts. our work shows that large language models are a fitting solution for knowledge graph construction, specifically where labelled data is scarce or the ontology dynamic.