Evaluating the Knowledge Base Completion Potential of GPT (2023)

Abstract: Structured knowledge bases (KBs) are an asset for search engines and other applications but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT’s potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, it provides solid improvements over earlier approaches with smaller LMs. In particular, we show that it is feasible to extend Wikidata by 27M facts at 90% precision.

## introduction structured knowledge bases (kbs) like wikidata @xcite , dbpedia @xcite , and @xcite are employed in many knowledge-centric applications like search, question answering and dialogue. constructing and completing these kbs at high quality and scale is a long-standing research challenge, and multiple benchmarks exist, e.g., fb15k @xcite , codex @xcite , and lm-kbc22 @xcite . text-extraction, knowledge graph embeddings, and lm-based knowledge extraction have continuously moved scores upwards on these tasks, and leaderboard portals like paperswithcode foot_0 provide evidence for that. recently, lms have been purported as a promising source of structured knowledge. starting from the seminal lama paper @xcite , a throve of works have explored how to better probe, train, or fine-tune these lms @xcite . nonetheless, we observe a certain divide between these late-breaking investigations, and practical kb completion. while recent lm-based approaches often focus on simple methodologies that produce fast results, practical kbc so far is a highly precision-oriented, extremely laborious process, involving a very high degree of manual labour, either for manually creating statements @xcite , or for building comprehensive scraping, cleaning, validation, and normalization pipelines @xcite . for example, part of yago's success stems from its @xmath2, and according to @xcite , the google knowledge vault was not deployed into production partly because it did not achieve 99% accuracy. yet, many previous lm analyses balance precision and recall or report precision/hits@k values, implicitly tuning systems towards balanced recall scores resulting in impractical precision. it is also important to keep in mind the scale of kbs: wikidata currently contains around 100 million entities and 1.2b statements. the cost of producing such kbs is massive. an estimate from 2018 sets the cost per statement at 2 @xmath0. in this paper, we conduct a systematic analysis of the kb completion potential of gpt, where we focus on high precision. we evaluate by employing (i) a recent kb completion benchmark, wd-known, @xcite , which randomly samples facts from wikidata and (ii) by a manual evaluation of subject-relation pairs without object values. our main results are: based on our analysis of 41 common relations, we would be able to add a total of 27m highaccuracy statements. an inherent design decision in kbc is the p/r trade-off -academic projects are often open to trade these freely (e.g., via f-1 scores), yet production environments are often critically concerned with precision, e.g., wikidata generally discouraging statistical inferences, and industrial players likely use to a considerable degree human editing and verification @xcite . for example in all of rescal, transe, and lama, the main results focus on metrics like hits@k, mrr, or auc, which provide no bounds on precision. lms for kb construction knowledge extraction from lms provides fresh hope for the synergy of automated approaches and high-precision curated kbs. it provides remarkably straightforward access to very large text corpora: the basic idea by @xcite is to just define one template per relation, then query the lm with subject-instantiated versions, and retain its top prediction(s). a range of follow-up works appeared, focusing, e.g., on investigating entities, improving updates, exploring storage limits, incorporating unique entity identifiers, and others @xcite @xcite @xcite @xcite @xcite . nonetheless, we observe the same gaps as above: the high-precision area, and completion of already existing resources, are not well investigated. several works have analyzed the potential of larger lms, specifically gpt-3 and gpt-4,. they investigate few-shot prompting for extracting factual knowledge for kbc @xcite or for making the factual knowledge in a lm more explicit @xcite . these models can aid in building a knowledge base on wikidata or improving the interpretability of lms. despite the variance in the precision of extracted facts from gpt-3, it can peak at over 90% for some relations. recently, gpt-4's capabilities for kbc and reasoning were examined @xcite . this research compared gpt-3, chatgpt, and gpt-4 on information extraction tasks, kbc, and kgbased question answering. however, these studies focus on popular statements from existing kbs, neglecting the challenge of introducing genuinely new knowledge in the long tail. in @xcite , we analyzed to which degree bert can complete the wikidata kb, i.e., provide novel statements. together with the focus on high precision, this is also the main difference of the present work to the works cited above, which evaluate on knowledge already existing in the kb, and do not estimate how much they could add. ## analysis method dataset we consider the 41 relations from the lama paper @xcite . for automated evaluation and threshold finding, we employ the wd-known dataset @xcite . unlike other kbc datasets, this one contains truly long- ## results and discussion can gpt models complete wikidata at precision and scale? in in absolute terms, we find a massive number of high-accuracy statements that could be added to the writtenin relation (18m), followed by spo-kenlanguage and nativelanguage (4m each). in relative terms, the additions could increase the existing relations by up to 1200%, though there is a surprising divergence (4 relations over 100%, 11 relations below 20%). does gpt provide a quantum leap? generating millions of novel high-precision facts is a significant achievement, though the manually verified precision is still below what industrial kbs aim for. the wide variance in relative gains also shows that gpt only shines in selected areas. in line with previous results @xcite , we find that gpt can do well on relations that exhibit high surface correlations (person names often give away their nationality), otherwise the task remains hard. in is this cost-effective? previous works have estimated the cost of kb statement construction at 1 ct. (highly automated infobox scraping) to @xmath1168,000, and yield approximately 27m novel statements. does "don't know" prompting help? in ## conclusion we provided the first analysis of the real kb completion potential of gpt. our findings indicate that gpt-3 could add novel knowledge to wikidata, at unprecedented scale and quality (27m statements at 90% precision). compared with other approaches the estimated cost of $168,000 is surprisingly cheap, and well within the reach of industrial players. we also find that, in the high-precision bracket, gpt-3 distills web content to a degree that context augmentation does not easily help. open issues remain in particular around identifying high-confidence predictions within an lm's generations @xcite @xcite , and the choice of examples.