A Re-evaluation of Knowledge Graph Completion Methods (2020)

Abstract: Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. A vast number of state-of-the-art KGC techniques have got published at top conferences in several research fields, including data mining, machine learning, and natural language processing. However, we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods. In this paper, we find that this can be attributed to the inappropriate evaluation protocol used by them and propose a simple evaluation protocol to address this problem. The proposed protocol is robust to handle bias in the model, which can substantially affect the final results. We conduct extensive experiments and report performance of several existing methods using our protocol. The reproducible code has been made publicly available.

## introduction real-world knowledge bases are usually expressed as multi-relational graphs, which are collections of factual triplets, where each triplet (h, r, t) represents a relation r between a head entity h and a tail entity t. however, real-word knowledge bases are usually incomplete @xcite , which motivates the research of automatically predicting missing links. a popular approach for knowledge graph completion (kgc) is to embed entities and relations into continuous vector or matrix space, and use a well-designed score function f (h, r, t) to measure the plausibility of the triplet (h, r, t). most of the previous methods use translation distance based @xcite @xcite and semantic matching based @xcite @xcite @xcite scoring functions which are easy to analyze. however, recently, a vast number of neural network-based methods have been proposed. they have complex score functions which utilize blackbox neural networks including convolutional neural networks (cnns) @xcite , recurrent neural networks (rnns) @xcite , graph neural networks (gnns) @xcite , and capsule networks @xcite . while some of them report state-of-the-art performance on several benchmark datasets that are competitive to previous embedding-based approaches, a considerable portion of recent neural network-based papers report very high performance gains which are not consistent across different datasets. moreover, most of these unusual behaviors are not at all analyzed. such a pattern has become prominent and is misleading the whole community. in this paper, we investigate this problem and find that this is attributed to the inappropriate evaluation protocol used by these approaches. we demonstrate that their evaluation protocol gives a perfect score to a model that always outputs a constant irrespective of the input. this has lead to artificial inflation of performance of several models. for this, we find a simple evaluation protocol that creates a fair comparison environment for all types of score functions. we conduct extensive experiments to re-examine some recent methods and fairly compare them with existing approaches. the source code of the paper has been publicly available at http://github.com/svjan5/kg-reeval . ## observations in this section, we first describe our observations and concerns and then investigate the reason behind. ## evaluation protocols for kgc in this section, we present different evaluation protocols that can be adopted in knowledge graph completion. we further show that inappropriate evaluation protocol is the key reason behind the unusual behavior of some recent nn-based methods. how to deal with the same scores? an essential aspect of the evaluation method is to decide how to break ties for triplets with the same score. more concretely, while scoring the candidate set t , if there are multiple triplets with the same score from the model, one should decide which triplet to pick. assuming that the triplets are sorted in a stable manner, we design a general evaluation scheme for kgc, which consists of the following three different protocols: ## experiments in this section, we conduct extensive experiments using our proposed evaluation protocols and make a fair comparison for several existing methods. ## conclusion in this paper, we performed an extensive reexamination study of recent neural network based kgc techniques. we find that many such models have issues with their score functions. combined with inappropriate evaluation protocol, such methods reported inflated performance. based on our observations, we propose random evaluation protocol that can clearly distinguish between these affected methods from others. we also strongly encourage the research community to follow the random evaluation protocol for all kgc evaluation purposes.