Knowledge Base Completion for Long-Tail Entities (2023)

Abstract: Despite their impressive scale, knowledge bases (KBs), such as Wikidata, still contain significant gaps. Language models (LMs) have been proposed as a source for filling these gaps. However, prior works have focused on prominent entities with rich coverage by LMs, neglecting the crucial case of long-tail entities. In this paper, we present a novel method for LM-based-KB completion that is specifically geared for facts about long-tail entities. The method leverages two different LMs in two stages: for candidate retrieval and for candidate verification and disambiguation. To evaluate our method and various baselines, we introduce a novel dataset, called MALT, rooted in Wikidata. Our method outperforms all baselines in F1, with major gains especially in recall.

## introduction motivation and problem. knowledge base completion (kbc) is crucial to continuously enhance the scope and scale of large knowledge graphs (kgs). it is often cast into a link prediction task: infer an o(bject) argument for a given s(ubject)-p(redicate) pair. however, the task is focused on the kg itself as the only input, and thus largely bound to predict spo facts that are also derivable by simple logical rules for inverse predicates, transitive predicates etc. @xcite . to obtain truly new facts, more recent methods tap into large language models (lms) that are learned from huge text collections, including all wikipedia articles, news articles and more. the most promising approaches to this end generate cloze questions for knowledge acquisition and ask lms to generate answers @xcite . the lm input is often augmented with carefully crafted short prompts (e.g., a relevant wikipedia paragraph) @xcite @xcite . however, notwithstanding great success for question answering to humans, the lm-based approach falls short on meeting the high quality requirements for enriching a kg with crisp spo facts. even if most answers are correct, there is a non-negligible fraction of false or even "hallucinated" outputs by the lm, and large kgs, like wikidata @xcite , cannot tolerate error rates above 10 percent. moreover, even correct answers are not properly canonicalized: they are surface phrases and not unique entities in the kg. these problems are further aggravated when the to-be-inferred o arguments are long-tail entities, with very few facts in wikidata. here, we call an entity long-tail when it has less than 14 triples in wikidata, because nearly 50% of the wikidata entities have fewer than 14 triples. these are exactly the pain point that calls for kbc. this paper addresses this problem. as an example, consider the late canadian singer lhasa de sela. wikidata solely covers basic biographic facts and selected awards, nothing about her music. however, text sources such as her wikipedia article or other web pages provide expressive statements about her albums, songs, collaborations etc. for example, we would like to spot the facts that ⟨lhasa de sela, collaboratedwith, bratsch⟩ and ⟨lhasa de sela, performedsong, anyone and everyone⟩. note that capturing these as spo facts faces the challenge of having to capture and disambiguate multi-word names ("lhasa de sela") and common-noun phrases ("anyone and everyone"). when trying to extract such statements via cloze questions or more refined prompts to lms such as gpt-3 @xcite or chatgpt, the outputs would often be "lhasa", which is highly ambiguous, or "everyone", which is incomplete and impossible to interpret. approach and contribution. this paper devises a novel method for knowledge base completion (kbc), specifically geared to cope with long-tail our method leverages transformer-based language models in a new way. most notably, we employ two different lms in a two-stage pipeline, as shown in figure 1 . the first stage generates candidate answers to input prompts and gives cues to retrieve informative sentences from wikipedia and other sources. the second stage validates (or falsifies) the candidates and disambiguates the retained answer strings onto entities in the underlying kg (e.g., mapping "lhasa" to lhasa de sela , and "bratsch" to bratsch (band)). the novel contributions of this work are the following: our code and data are available at https:// github.com/tigerchen52/long_tail_kbc . candidate generation. we devise a generic prompt template for cloze questions, in order to infer an "o" answer for a given s-p pair. this merely requires a simple verbalizer for the relation p: ## malt: new dataset for benchmarking benchmarks for kbc and lm-as-kb cover facts for all kinds of entities, but tend to focus on prominent ones with frequent mentions. likewise, benchmarks for relation extraction (re) from text, most notably tacred @xcite , docred @xcite and lama @xcite do not reflect the difficulty of coping with long-tail entities and the amplified issue of surface-name ambiguity (see relation id candidate generation corroboration and canonicalization founded by p112 the business [x] is founded by which person? the business [x] is founded by [ent] this person [ent] performer p175 the song [x] is performed by which person? the song [x] is performed by [ent] this person [ent] composer p86 the song [x] is composed by which person? the song [x] is composed by [ent] this person [ent] place of birth p19 the person [x] was born in which place? the person [x] was born in [ent] this place [ent] place of death p20 the person [x] died in which place? the person [x] died in [ent] this place [ent] employer p108 the person [x] worked in which place? the person [x] worked in [ent] this place [ent] educated at p69 the person [x] graduated from which place? the person [x] graduated from [ent] this place [ent] residence p551 the person [x] lived in which place? the person [x] lived in [ent] this place [ent] to construct the dataset, we focus on three types of entities: business, musiccomposition and human, richly covered in wikidata and often involving long-tail entities. we randomly select subjects from the respective relations in wikidata, and keep all objects for them. we select a total of 8 predicates for the 3 types; the dataset contains 65.3% triple facts where the o entity is a multi-word phrase, and 58.6% ambiguous facts where the s or o entities share identical alias names in wikidata. for example, the two ambiguous entities ,"birmingham, west midlands (q2256)" and "birmingham, alabama (q79867)", have the same label value "birmingham". in total, 87.0% of the sample facts have s entities in the long tail, where we define long-tail entities to have at most 13 wikidata triples. ## experimental evaluation baselines. to the best of our knowledge, there is no prior work on kbc or lm-as-kb that is specifically geared for coping with long-tail entities. as a proxy, we thus compare to several state-of-theart methods for relation extraction (re) from text. at test time, these methods receive the retrieved wikipedia sentences for a ground-truth spo fact and the sp pair as input, and are run to extract the withheld o argument (sentence-level extraction). we compare to the following baselines: setup. there are two hyper-parameters for all competitors, the number of candidates k (or the "top-k" hyper-parameter for baseline models) and the threshold α for cutting off the extracted triples. for our framework, k is 20 for all competitors and the threshold α is learned by using a hold-out (20%) validation set. we report results for precision, recall and f1, with the original wikidata triples as ground truth. although malt provides canonicalized entities, we consider the extracted o to be a correct prediction as long as it appears in the alias table because some baselines themselves cannot do disambiguation. our method is completely unsupervised, and the only additional cost is prompt. we manually design one template for each relation (as shown in our method still leaves substantial room for further improvement, underlining the challenging nature of inferring facts for long-tail entities. we think of our method as a building block to aid a human curator by judicious suggestions for facts that would augment the kg. many of the inferred spo facts are indeed completely missing in wikidata; so they are also not in the withheld ground-truth samples for the above evaluation. to estimate how many facts we could potentially add to the kg and how good our automatically inferred predictions are, we picked 25 samples for each relation, a total of 250 fact candidates, and asked human annotators to assess their correctness. over all relations, this achieved an average precision of 61%. for the relation educated at, our method even has 76% precision, and this is a case where the kg has enormous gaps: out of 10m sampled entities of type human, only 65% have facts for this relation. for this case, our kbc method collected 1.2m candidate facts, showing the great potential towards closing these gaps. ## conclusion we highlighted the challenge of knowledge base completion (kbc) for long-tail entities, introduced the malt dataset for experimental comparisons and fostering further research, and presented a completely unsupervised method for augmenting knowledge bases with long-tail facts. our method operates in two stages, candidate generation and candidate corroboration (incl. disambiguation), and leverages two different lms in a complementary way. experimental results show substantial gains over state-of-the-art baselines, and highlight the benefits of our two-stage design with two lms complementing each other.