I nt KB : A Verifiable Interactive Framework for Knowledge Base Completion (2020)

Abstract: Knowledge bases (KBs) are essential for many downstream NLP tasks, yet their prime shortcoming is that they are often incomplete. State-of-the-art frameworks for KB completion often lack sufficient accuracy to work fully automated without human supervision. As a remedy, we propose : a novel interactive framework for KB completion from text based on a question answering pipeline. Our framework is tailored to the specific needs of a human-in-the-loop paradigm: (i) We generate facts that are aligned with text snippets and are thus immediately verifiable by humans. (ii) Our system is designed such that it continuously learns during the KB completion task and, therefore, significantly improves its performance upon initial zero- and few-shot relations over time. (iii) We only trigger human interactions when there is enough information for a correct prediction. Therefore, we train our system with negative examples and a fold-option if there is no answer. Our framework yields a favorable performance: it achieves a hit@1 ratio of 29.7% for initially unseen relations, upon which it gradually improves to 46.2%.

## introduction knowledge bases (kbs) present databases that store information about entities and the relations among them. prominent examples are, for instance, wikidata @xcite , freebase @xcite , nell @xcite , dbpedia @xcite , or @xcite . owed to their structured information, kbs are widely used in various downstream tasks of nlp such as, e. g., entity linking @xcite , query expansion @xcite , co-reference resolution @xcite , and question answering @xcite . yet the performance of downstream nlp tasks is often impeded due to the issue that kbs are incomplete. for example, for newspapers, wikidata lists the place of publication for only 37% of them, while the owner is stored for only a staggering 9%. hence, there is a need to develop scalable strategies for kb completion. prior work on kb completion can be divided into two paradigms. on the one hand, kb completion via link prediction. here the structural information of an existing, yet incomplete kb is leveraged in order to predict new facts @xcite @xcite . on the other hand, kb completion can be built upon a text corpus, so that new facts are extracted directly from the narrative materials @xcite . approaches belonging to the latter paradigm are scarce, as they usually rely upon some way of aggregation information from multiple, different corpora (where credible additions to a kb are signaled by cross-corpus appearances). in contrast, we develop a framework for kb completion over a single corpus, where such verification is obtained in an interactive manner. knowledge base completion is often designed as an automated task where predictions are directly integrated into an existing but incomplete kb. this can be problematic for various reasons: (i) the performance of state-of-the-art systems is not of sufficient accuracy, so that it allows for an automatic integration of new facts @xcite . hence, manual verification is still necessary in practice. (ii) predictions in kb completion often involve zero-or few-shot settings (e. g., relations for which there is none or only or few examples available during training), this further diminishes their performance @xcite . (iii) state-of-the-art systems do not generate facts that are verifiable by humans because their predictions are mainly based on structural similarity within graphs. thus, it is impossible for humans to manually detect false-positive predictions. this problem becomes especially critical in domains as, e.g., medicine, where kbs serve as the basis for high-stake decisions. to this end, kb completion should allow for simple human verification. yet, to the best of our knowledge, research on verifiable kb completion is lacking. in this paper, we propose intkb: a novel interactive framework for kb completion based on question answering (qa). unlike previous work, we design our framework to fulfill a human-in-the-loop paradigm. in intkb, new candidate facts are presented to human annotators, who can then approve or reject the candidates before they are integrated into the kb. based on this setting, we overcome several challenges that are inherent to prior work: first, candidate predictions need to be human verifiable. it is not sufficient to show human annotators simply a new kb entry, as it would require a substantial effort to verify its correctness. therefore, we align predictions with text snippets that make them directly verifiable. second, in order to tackle the low performance in cold-start (e.g., zero-shot) settings, we design our framework in such a way that it gradually improves over time from past human interactions. here human-approved predictions are fed back into our training framework. third, intkb is designed for an interactive use with the aim of requiring only little manual effort. for this, it only triggers human interactions when there is sufficient evidence (i. e., the system can decide to fold otherwise). our framework is realized on the basis of a question answering (qa) pipeline. the prime benefit of this is that it operates on only a single text corpus (rather than multiple) when generating new facts with high confidence. in addition, we develop a training framework based on knowledge transfer via fact alignment and continuous updates. it allows us to make predictions in cold-start settings. it also achieves a favorable precision when predicting long-tail relations that are rare. we evaluate our system in an extensive set of experiments. overall, we annotated around 3000 new facts from human interactions. we demonstrate that our framework continuously learns from those interactions: over time, the hit@1 ratio on initially unseen zero-shot relations increases from 29.7% to over 46.2% . a live demo is available from https://wikidatacomplete.org , the source code is available from https://github.com/bernhard2202/intkb . ## related work recent research on kb completion can be divided into two main paradigms: (i) kb completion via link prediction based on the graph structure of an existing yet incomplete kb and (ii) kb completion using free text from the web or a corpus. both are detailed in the following. ## preliminaries knowledge base: a knowledge base is given by a tuple v, e , where v is a set of entities and e is a set of relations between them (i. e., so-called "facts"). relations themselves are triplets h, r j , @xmath0, where @xmath1, @xmath2, and r j is the relation type. relationships in our setting are specified by one (or multiple) names [ω (j) 0 , . . .] denoting their semantic meaning. for instance, the relationship referring to a company owner could be given by r 5 : ["is owned by", "belongs to", . . .]. similarly, all entities @xmath3, . . .]. an example for the new york times would be ["nyt", "ny times", "new york times", . . . ]. text corpus: in addition, we are given a text corpus @xmath4, . . . d n ], where d i refers to the i-th document. documents are associated with entities and, to this end, we denote the document associated with entity @xmath5. in our experiments, this corresponds to the wikipedia page that describes an entity. ## intkb: a verifiable framework for interactive kb completion this section describes our general framework (see fig. 1 ) for interactive kb completion over free text. the input to our system is a query consisting of a head-entity and relation-type. we then wish to annotate the answer to a given query should be presented and, in addition, a textual source where the answer was extracted from. this makes our kb completion verifiable by humans. the ground-truth tail entity (see sec. 4.1). for this, we customize a qa pipeline. the qa pipeline comprises of three modules for sentence selection, relation extraction, and answer re-ranking (see sec. 4.2). finally, we use a module for answer-triggering that decides on whether to invoke a human interaction or end the prediction process by folding (see sec. 4.3). in the case that a human interaction is triggered, we show the new candidate fact to human annotators and they can either approve or decline it. for approved predictions, we link the textual mention back to the actual entity (see sec. 4.4) and add the new fact to the kb. finally, we use the feedback from the human annotators to continuously update our framework (see sec. 5). ## continuous learning for kb completion this section explains how we learn the components inside our intkb framework during (1) initialization as part of a cold-start and (2) continuous improvements from user feedback. ## dataset common datasets for kb completion @xcite are designed for link prediction and lack a text corpus from which the task of kb completion could be performed. the wikireadings dataset @xcite comes close to our needs but 99% of the dataset are covered by only 180 relations. hence, it largely lacks infrequent long-tail relations that are known to be notoriously hard to predict. instead, we construct a dataset that is tailored to our setting by following best-practice as described below (and in the supplements). our dataset is a combination of (i) a incomplete kb in the form of a subset of wikidata and (ii) a text corpus, namely the english wikipedia. both are detailed below. kb subsets: relations in the kb are split into two distinct subsets (for performing different evaluations later): (i) known relations represent an initially known and incomplete kb, while (ii) zero-shot or cold-start relations are unseen during the initial training. both subsets are further divided into a training set and a hold-out set for testing. the different subsets are used at different states of our training and evaluation procedure as follows: the known training-set is used during the initial fact alignment and knowledge transfer in order to learn our system. the known test-set is used to evaluate the performance gains on known relations when using both fact alignment and knowledge transfer from before. the zero-shot training-set is used for simulating user interactions. we remind that, even though this set is called "train", we do not use its ground-truth labels directly but simulate user interactions that approve correct labels and reject incorrect labels. the zero-shot test-set is used to evaluate performance on those unseen relations. it allows us to quantify the performance improvements from continuous learning. negative examples: we synthetically generate unanswerable examples in order to make the prediction task harder and evaluate the answer triggering module. therefore, we copy queries in our test-sets and pair them with wikipedia articles in which all sentences that contain mentions of the correct tail entity have been removed. ## computational experiments overview: our proposed training framework is evaluated as follows: in sec. 7.1, we first record the performance on the known subset of the dataset, i. e., on those relations that were initially present in our kb. in sec. 7.2, we evaluate the performance on the zero-shot subset of our dataset. this represents the initial cold-start performance of our system. furthermore, in sec. 7.3, we present our experiment on continuous improvements from over 9000 simulated user interactions. finally, in sec. 7.4, we present our results including entity back-linking. performance metrics: in the following, our kb completion over free text is evaluated using different performance metrics. we report the hit rate at k, i. e., h@k, when evaluating only on positive examples (e.g., queries that have an answer in the text corpus and hence disabling the answer triggering module for this evaluation). for negative examples (e.g., only queries that are not answerable), we report the negative recall neg-rec, which is calculated from the fraction of negative samples where the framework triggered the fold option. finally, we report the combined accuracy comb-acc when evaluating on positive and negative samples at the same time. the combined accuracy is 1 for an example iff it was positive and the correct answer is predicted or it was negative and the fold option was triggered, and 0 otherwise. baselines: our proposed kb completion is compared against the following two baselines: (i) bert-sentence: this baselines combines the relation extraction module from bert with our proposed sentence selection. thereby, it resembles the system from @xcite in the sense that we replaced the machine comprehension model by bert and the manual question templates replaced by our (noisy) keyword queries. (ii) naïve qa pipeline: this baseline resembles our pipeline for kb completion including the module for answer re-ranking. however, all modules were simply trained on the squadv2 dataset @xcite , which includes negative examples. other baselines lack comparability as they operate on structural information in the kb and not textual content. this is to the best of our knowledge the first verifiable kb completion framework. ## discussion this paper contributes kb completion using only a single text corpus. for this purpose, we developed a combination of a neural qa pipeline and a novel training framework with fact alignment and contionus updates. based on it, new facts for incomplete kbs can be inferred effectively. our training framework provides a natural approach to align facts with sentences from the corpus and, therefore, makes all prediction verifiable. this is a clear advantage over most of the current kb completion techniques that lack means for new facts to be verified by humans. in practice, it has two main implications: (i) facts can be shown along textual evidence to kb editors and thereby easily approved or rejected. this thus facilitates a human-in-the-loop framework. (ii) in our work, textual evidence of the proposed facts provides information on the provenance, i. e., the source of facts can be traced. this is important in domains (e. g., medicine), where kbs serve as the basis for high-stakes decisions.