C o DE x: A C omprehensive K nowledge G raph C ompletion B enchmark (2020)

Abstract: We present CoDEx, a set of knowledge graph completion datasets extracted from Wikidata and Wikipedia that improve upon existing knowledge graph completion benchmarks in scope and level of difficulty. In terms of scope, CoDEx comprises three knowledge graphs varying in size and structure, multilingual descriptions of entities and relations, and tens of thousands of hard negative triples that are plausible but verified to be false. To characterize CoDEx, we contribute thorough empirical analyses and benchmarking experiments. First, we analyze each CoDEx dataset in terms of logical relation patterns. Next, we report baseline link prediction and triple classification results on CoDEx for five extensively tuned embedding models. Finally, we differentiate CoDEx from the popular FB15K-237 knowledge graph completion dataset by showing that CoDEx covers more diverse and interpretable content, and is a more difficult link prediction benchmark. Data, code, and pretrained models are available at https://bit.ly/2EPbrJs .

## introduction knowledge graphs are multi-relational graphs that express facts about the world by connecting entities (people, places, things, concepts) via different types of relationships. the field of automatic knowledge graph completion (kgc), which is motivated by the fact that knowledge graphs are usually incomplete, is an active research direction spanning several subfields of artificial intelligence @xcite @xcite . as progress in artificial intelligence depends heavily on data, a relevant and high-quality benchmark is imperative to evaluating and advancing the state of the art in kgc. however, the field has largely remained static in this regard over the past decade. outdated subsets of freebase @xcite are most commonly used for evaluation in kgc, even though freebase had known quality issues @xcite and was eventually deprecated in favor of the more recent wikidata knowledge base @xcite . indeed, kgc benchmarks extracted from freebase like fb15k and fb15k-237 @xcite are questionable in quality. for example, fb15k was shown to have train/test leakage @xcite . later in this paper ( § 6.2), we will show that a relatively large proportion of relations in fb15k-237 can be covered by a trivial frequency rule. to address the need for a solid benchmark in kgc, we present codex, a set of knowledge graph completion datasets extracted from wikidata and its sister project wikipedia. inasmuch as wikidata is considered the successor of freebase, codex improves upon existing freebase-based kgc benchmarks in terms of scope and level of difficulty ( foundations we survey evaluation datasets in encyclopedic knowledge graph completion to motivate a new benchmark ( § 2 and appendix a). data we introduce codex, a benchmark consisting of three knowledge graphs varying in size and structure, entity types, multilingual labels and descriptions, and-unique to codex-manually verified hard negative triples ( § 3). to better understand codex, we analyze the logical relation patterns in each of its datasets ( § 4). benchmarking we conduct large-scale model selection and benchmarking experiments, reporting baseline link prediction and triple classification results on codex for five widely used embedding models from different architectural classes ( § 5). ## existing datasets we begin by surveying existing kgc benchmarks. note that we focus on data rather than models, so we only overview relevant evaluation benchmarks here. for more on existing kgc models, both neural and symbolic, we refer the reader to @xcite and @xcite . ## data collection in this section we describe the pipeline used to construct codex. for reference, we define a knowledge graph g as a multi-relational graph consisting of a set of entities e, relations r, and factual statements in the form of (head, relation, tail) triples (h, r, t) ∈ e × r × e. ## analysis of relation patterns to give an idea of the types of reasoning necessary for models to perform well on codex, we analyze the presence of learnable binary relation patterns within codex. the three main types of such patterns in knowledge graphs are symmetry, inversion, and compositionality @xcite . we address symmetry and compositionality here, and omit inversion because we specifically removed inverse relations to avoid train/test leakage ( § 3.2). ## benchmarking next, we benchmark performance on codex for the tasks of link prediction and triple classification. to ensure that models are fairly and accurately compared, we follow @xcite , who conducted what is (to the best of our knowledge) the largest-scale hyperparameter tuning study of knowledge graph embeddings to date. note that codex can be used to evaluate any type of kgc method. however, we focus on embeddings in this section due to their widespread usage in modern nlp @xcite . ## comparative case study finally, we conduct a comparative analysis between codex-m and fb15k-237 ( § 2.1) to demonstrate the unique value of codex. we choose fb15k-237 because it is the most popular encyclopedic kgc benchmark after fb15k, which was already shown to be an easy dataset by @xcite . we choose codex-m because it is the closest in size to fb15k-237. ## conclusion and outlook we present codex, a set of knowledge graph completion datasets extracted from wikidata and wikipedia, and show that codex is suitable for multiple kgc tasks. we release data, code, and pretrained models for use by the community at https://bit.ly/2epbrjs . some promising future directions on codex include: overall, we hope that codex will provide a boost to research in kgc, which will in turn impact many other fields of artificial intelligence.