Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction (2020)

Abstract: Open Information Extraction systems extract (“subject text”, “relation text”, “object text”) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task,i.e., predicting test facts by completing (“subject text”, “relation text”, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. We performed experiments with a prototypical knowledge graph embedding model for openlink prediction. While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.

## introduction a knowledge graph (kg) @xcite ) is a set of (subject, relation, object)-triples, where the subject and object correspond to vertices, and relations to labeled edges. in curated kgs, each triple is fully disambiguated against a fixed vocabulary of entities 1 and relations. an application for kgs, for example, is the problem of drug discovery based on bio-medical knowledge @xcite . the construction of a curated bio-medical kg, which is required for 1 for brevity, "entities" denotes both entities (e.g. prince) and concepts (e.g. musician) throughout the paper. such an approach, is challenging and constrained by the available amount of human effort and domain expertise. many tools that could assist humans in kg construction (e.g., an entity linker) need a kg to begin with. moreover, current methods for kg construction often rely on the rich structure of wikipedia, such as links and infoboxes, which are not available for every domain. therefore, we ask if it is possible to make predictions about, for example, new drug applications from raw text without the intermediate step of kg construction. open information extraction systems (oie) @xcite automatically extract ("subject text", "relation text", "object text")-triples from unstructured data such as text. we can view oie data as an open knowledge graph (okg) @xcite , in which vertices correspond to mentions of entities and edges to open relations (see fig. 1 ). our overarching interest is whether and how we can reason over an okg without any canonicalization and without any supervision on its latent factual knowledge. the focus of this study are the challenges of benchmarking the inference abilities of models in such a setup. a common task that requires reasoning over a kg is link prediction (lp). the goal of lp is to predict missing facts in a kg. in general, lp is defined as answering questions such as (nbc, headquar-terin, ?) or (?, headquarterin, newyorkcity); see fig. 2a . in okgs, we define open link prediction (olp) as follows: given an okg and a question consisting of an entity mention and an open relation, predict mentions as answers. a predicted mention is correct if it is a mention of the correct answer entity. for example, given the question ("nbc-tv", "has office in", ?), correct answers include "nyc" and "new york"; see fig. 2b ). to evaluate lp performance, the lp model is trained on known facts and evaluated to predict unknown facts, i.e., facts not seen during training. a simple but problematic way to transfer this approach to okgs is to sample a set of evaluation triples from the okg and to use the remaining part of the okg for training. to see why this approach is problematic, consider the test triple ("nbc-tv", "has office in", "new york") and suppose that the triple ("nbc", "has headquarter in", "nyc") is also part of the okg. the latter triple essentially leaks the test fact. if we do not remove such facts from the training data, a successful models only paraphrases known facts but does not perform reasoning, i.e., does not predict genuinely new facts. furthermore, we also want to quantify if there are other trivial explanations for the prediction of an evaluation fact. for example, how much can be predicted with simple popularity statistics, i.e., only the mention, e.g. ("nbc-tv", ?), or only the relation, e.g. ("has office in", ?). such non-relational information also does not require reasoning over the graph. to experimentally explore whether it is possible to predict new facts, we focus on knowledge graph embedding (kge) models @xcite , which have been applied successfully to lp in kgs. such models can be easily extended to handle the surface forms of mentions and open relations. our contributions are as follows: we propose the olp task, an olp evaluation protocol, and a method to create an olp benchmark dataset. using the latter method, we created a large olp benchmark called olpbench, which was derived from the state-of-the-art oie corpus opiec @xcite . olpbench contains 30m open triples, 1m distinct open relations and 2.5m distinct mentions of approximately 800k entities. we investigate the effect of paraphrasing and nonrelational information on the performance of a prototypical kge model for olp. we also investigate the influence of entity knowledge during model selection with different types of validation data. for training kge models on such large datasets, we describe an efficient training method. in our experiments, we found the olp task and olpbench to be very challenging. still, the kge model we considered was able to predict genuinely new facts. we also show that paraphrasing and non-relational information can indeed dilute performance evaluation, but can be remedied by appropriate dataset construction and experimental settings. ## open knowledge graphs okgs can be constructed in a fully automatic way. they are open in that they do not require a vocabulary of entities and relations. for this reason, they can capture more information than curated kgs. for example, different entity mentions can refer to different versions of an entity at different points of time, e.g., "senator barack obama" and "president barack obama". similarly, relations may be of varying specificity: headquarterin may be expressed directly by open relations such as "be based in" or "operate from" but may also be implied by "relocated their offices to". in contrast to kgs, okgs contain rich conceptual knowledge. for example, the triple ("a class action lawsuit", "is brought by", "shareholders") does not directly encode entity knowledge, although it does provide information about entities that link to "a class action lawsuit" or "shareholders". okgs tend to be noisier and the factual knowledge is less certain than in a kg, however. they ## open link prediction the open link prediction task is based on the link prediction task for kgs @xcite , which we describe first. let e be a set of entities, r be a set of relations, and t ⊆ e × r × e be a knowledge graph. consider questions of the form q @xmath0, k, j) or q @xmath1, k, ?), where i, @xmath2, respectively, and @xmath3. the link prediction problem is to provide answers that are correct but not yet present in t . in okgs, only mentions of entities and open relations are observed. we model each entity mention and each open relation as a non-empty sequence of tokens from some vocabulary v (e.g., a set of words). denote by @xmath4. an open knowledge graph t ⊂ m×m×m consists of triples of form (i, k, j), where i, @xmath5, resp., and @xmath6. note that we overload notation for readability: i, j, and k refer to entity mentions and open relations in okgs, but to disambiguated entities and relations in kgs. the intended meaning will always be clear from the context. we denote by m(e) and m(r) the sets of entity and relations present in t , respectively. the open link prediction task is to predict new and correct answers to questions (i, k, ?) or (?, k, j). answers are taken from m(e), whereas questions may refer to arbitrary mentions of entities and open relations from m. for example, for the question ("nbc-tv", "has office in", ?), we expect an answer from the set of mentions {"new york", "nyc", . . . } of the entity newyorkcity. informally, an answer (i, k, j) is correct if there is a correct triple (e 1 , r, e 2 ), where e 1 and e 2 are entities and r is a relation, such that i,j, and k are mentions of e 1 , e 2 , and r, respectively. ## creating the open link prediction benchmark olpbench an olp benchmark should enable us to evaluate a model's capability to predict genuinely new facts, i.e., facts can not be trivially derived. due to the nature of okgs, paraphrasing of facts may leak facts from validation and test data into training, making the prediction of such evaluation facts trivial. nevertheless, the creation of training and validation data should require as little human effort as possible so that the methodology can be readily applied to new domains. our mention-ranking protocol uses knowledge about entities for disambiguation (of the evaluation data, not the training data), however, which requires human effort to create. we investigate experimentally to what extent this entity knowledge is necessary for model selection and, in turn, how much manual effort is required to create a suitable validation dataset. in the following, we describe the source dataset of olpbench and discuss how we addressed the points above to create evaluation and training data. ## experiments in our experimental study, we investigated whether a simple prototypical olp model can predict genuinely new facts or if many successful predictions can be trivially explained by leakage or nonrelational information. our goal was to study the effectiveness and necessity of the mention-ranking protocol and leakage removal, and how much human effort is necessary to create suitable validation data. finally, we inspected data and model quality. we first describe the models and their training, then the performance metrics, and finally the evaluation. in our experimental results, model performance dropped by ≈25% with thorough leakage removal so that leakage due to paraphrasing is indeed a concern. we also implemented two diagnostic models that use non-relational infor-mation (only parts of a triple) to predict answers. these models reached ≈20-25% of the prototypical model's performance, which indicates that relational modelling is important. in our quality and error analysis, we found that at least 74% of the prediction errors were not due to noisy data. a majority of incorrectly predicted entity mentions have a type similar to the one of the true entity. ## conclusion we proposed the olp task and a method to create an olp benchmark. we created the large olp benchmark olpbench, which will be made publicly available foot_1 . we investigated the effect of leakage of evaluation facts, non-relational information, and entity-knowledge during model selection using a prototypical open link prediction model. our results indicate that most predicted true facts are genuinely new.