Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion (2021)

Abstract: We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, instead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation. In experiments, we curate two settings of InferWiki varying in sizes and structures, and apply the construction process on CoDEx as comparative datasets. The results and empirical analyses demonstrate the necessity and high-quality of InferWiki. Nevertheless, the performance gap among various inferential assumptions and patterns presents the difficulty and inspires future research direction. Our datasets can be found in https://github.com/TaoMiner/inferwiki .

## introduction knowledge graph completion (kgc) aims to predict missing links in kg by inferring new knowledge from existing ones. attributed to its reasoning ability, kgc models are crucial in alleviating the kg's incompleteness issue and benefiting many downstream applications, such as recommendation @xcite and information extraction @xcite . however, the kgc performance on existing benchmarks are still unsatisfactory -0.51 hit ratio@1 and 187 mean rank of the top-ranked model @xcite on the widely used fb15k237 @xcite . do we have a slow progress of head predicate tail test david location ? (ans: florida) train david placeofbirth atlanta david nationality u.s.a. test zurich travelmonth ? (ans: october) train zurich travelmonth jan., feb., mar., apr., may., jun., jul., aug., sep., nov., dec. in this paper, we re-think the task of kgc and construct a new benchmark dubbed inferwiki that highlights three fundamental objectives: test triples should be inferential: this is the essential requirement of kgc. each test triple should have supportive samples in the train set. however, we observe two major issues of current kgc datasets: unpredictable and meaningless test triples, which may hinder evaluating and advancing stateof-the-arts. as shown in test triples may be inferred positive, negative, or unknown. following open-world assumption: what is not observed in kg is not necessar- inference has various patterns. concentrating on limited patterns in evaluation may bring in severe bias. domain-specific datasets kinship @xcite and country @xcite only focus on a few relations and are nearly solved @xcite . general-domain wn18rr @xcite ) contains prevalent symmetry relation types, which incorrectly boosts the performance of rotate @xcite . clearly, limited patterns leads to unfair comparisons among kgc models. to this end, we curated an inferential kgc dataset extracted from wikidata and establish the benchmark with two settings of varying in sizes and structures: inferwiki64k and inferwiki16k. instead of random split, we mine rules via any-burl @xcite to guide train/test generation. all test triples are thus guaranteed inferential from training data. to avoid the rule leakage, we utilize two sets of triples: a large set for high-quality rule extraction and a small set for train/test split. moreover, we infer unseen triples and manually annotate them with positive, negative and unknown labels to improve the difficulty of evaluation following both closed-world and openworld assumptions. for inference patterns, we include and balance triples with different reasoning path length, relation types and patterns (e.g., symmetry and composition). our contributions can be summarized as follows: ## related work we can roughly classify current kgc datasets into two groups: inferential and non-inferential datasets. the first group is usually manually curated to ensure each testing sample can be inferred from training data through reasoning paths, while they only focus on specific relations, such as families @xcite , kinship @xcite , and country @xcite . the limited scale and inference patterns make them not challenging. hole @xcite achieves 99.7% acu-pr on the dataset of country. the second group of datasets are automatically derived from public kgs and randomly split positive triples into train/test, leading to a risk of testing samples non-inferential from training data. popular datasets include fb15k-237 @xcite , wn18rr @xcite . codex @xcite argues the scope and difficulty of the above datasets, thus propose a comprehensive dataset with manually verified hard negatives. in fact, inference is an important ability for intelligence. various fields study how inference is done in practice, ranging from logic to cognitive psychology. inference helps people make reliable predictions, which is also an expected ability for ai models. indeed, once deployed, a model may have to make a prediction when there is no evidence in the training set. but, instead of an unreliable guess, we highlight the ability to know unknown, a.k.a. open-world assumption. therefore, we aim to curate an large-scale inferential benchmark inferwiki including various inference patterns and testing samples (i.e., positive, negative, and unknown), for better evaluation. we list the statistics in ## dataset design we describe our dataset construction that comprises four steps: data preprocessing, rule mining, ruleguided train/test generation, and inferred test labeling. we then give a detailed analysis. ## case study of relation types we illustrate the most frequent relation types and their distribution of inferwiki64k and infer-wiki16k in figure 8 . we can see that inferwiki has a diverse relation types that are not limited to specific domains. besides, the triples of each relation type are well balanced. ## conclusion we highlighted three principles for kgc datasets: inferential ability, assumptions, and patterns, and contribute a large-scale dataset inferwiki. we established a benchmark with three types of seven kgc models on two tasks of triple classification and link prediction. the results present a detailed analysis regarding various inference patterns, which demonstrates the necessity of an inferential guarantee for better evaluation and the difficulty of new open-world triple classification. in the future, we are interested in cross-kgs inference and transfer @xcite , and investigating how to inject knowledge into deep learning architectures, such as for information extraction @xcite ) or text generation @xcite .