Entity Quality Enhancement in Knowledge Graphs through LLM -based Question Answering (2025)

Abstract: Most models for triple extraction from texts primarily focus on named entities. However, real-world applications often comprise non-named entities that pose serious challenges for entity linking and disambiguation. We focus on these entities and propose the first LLM-based entity revision framework to improve the quality of extracted triples via a multi-choice question-answering mechanism. When evaluated on two benchmark datasets, our results show a significant improvement, thereby generating more reliable triples for knowledge graphs.

## introduction triple extraction (te) is a well-established nlp task where several deep learning models @xcite @xcite @xcite , and more recently, llms @xcite @xcite have successfully been employed in benchmark datasets in different domains and languages (e.g., semeval-2010 task 8 @xcite , tacred @xcite , biored @xcite . most relation extraction models focus primarily on named entities such as person names, locations, and organizations, making them fail in dealing with a richer array of complex, non-named entities (hearafter n-ne). according to @xcite , n-ne are defined as noun phrases (nps) that can be the subject or object of a predicate within a sentence such as "decision list" and "parsing-based ne rules" in figure 1 . n-ne can have several forms ranging from nominal group (e.g., year 1944), containing adjectives and adverbs (e.g., very good questions), prepositional phrases (e.g., in the arab world), relative clauses or more complex syntactic constructions (e.g., near-term growth prospects of the global economy). n-ne are relatively frequent in textual data. for example, when manually analysing around 2k nps extracted from wikipedia, @xcite found that 78% of np heads are n-ne among which 38% are modified by an adjective, and 34% have a preposition. despite their importance, their frequency in popular benchmark datasets is relatively low (e.g., tacred only involves named entities). n-ne pose serious challenges in knowledge graph (kg) construction and reasoning, because they remain silent with no chance to be linked into an existing knowledge bases (kbs) such as yago4 @xcite or wikidata @xcite . figure 1 illustrates the impact these entities have on triple extraction from a sentence taken for semeval 2018 task 7. we compare the outputs of falcon-2 @xcite , a entity and relation linking tool over wikidata, and zero-shot gpt-4 against the gold label. although both models successfully identified the boundaries of the entities, they failed to correctly extract both the head and tail entities together. n-ne have received little attention in the literature. among the few works, open information extraction tools such as openie @xcite (see @xcite for a survey) output triples of subject, predicate, and object in an unsupervised way relaying on dependency parsers where relation arguments can contain n-ne. @xcite performed a qualitative manual study of the nature of n-ne in wikipedia. in this paper, we go one step further by proposing, for the first time as far as we know, an end to end llm-based entity revision framework that (a) automatically extracts triples from raw texts, (b) identifies n-ne, (c) enhances their quality by augmenting their likelihood of being successfully linked to an external knowledge base, which is a first important step to overall kg quality assessment @xcite . to this end, we adopt a multiple choice prompting (mcp) strategy on top of a triple extractor to verify the extracted entities. mcp has been successfully used as a self-evaluation method to mitigate llms errors in complex problems like arithmetic and commonsense reasoning @xcite @xcite . it is newly employed here for entity quality enhancement. our contributions are as follows: this paper is organized as follows. section 2 presents our overall framework, section 3 details the datasets used for evaluation, the experimental settings and evaluation metrics. we finally gives our results together with an error analysis in section 4. ## entity revision through llm-based question answering figure 3 shows our three-steps framework: (1) it first extracts triples using an in-context learning approach. (2) it then ranks candidate entities and (3) refines entity selection through a multiple-choice format to improve accuracy by learning from common extraction errors. it is important to note that our framework has been designed with modularity in mind, independently from the method used for triple extraction and how n-ne are initially identified. however as a first step and in order to evaluate the effectiveness of our approach when evaluated on benchmark datasets, we experiment with target relations as input to step 1, the subsequent steps are agnostic to this guidance. this allows to increase the number of matching triples generated by llms when compared to gold annotations (see below) and therefore ensure a sufficient number of instances to derive meaningful conclusions (see section 3.3 about the evaluation protocols). we detail below each step. ## conclusion in this paper, we explore the potential of llms incontext learning for entity revision. to address the challenges posed by non-named entities, we introduced a multiple-choice question-answering framework that revises extracted entities from llms while increasing their linking coverage with the largest open knowledge base. when evaluated on two benchmark relation extraction datasets, our results demonstrate the effectiveness of our framework. we believe our work is a first important step to account for non-named entities in knowledge graph construction. in this work, we apply a limited set of prompting techniques (zero-shot and few-shot in-context learning), which can be further explored in future research. we will also consider how improved entities affect downstream applications like question answering over knowledge graphs.