E valuating the C alibration of K nowledge G raph E mbeddings for T rustworthy L ink P rediction (2020)

Abstract: Little is known about the trustworthiness of predictions made by knowledge graph embedding (KGE) models. In this paper we take initial steps toward this direction by investigating the calibration of KGE models, or the extent to which they output confidence scores that reflect the expected correctness of predicted knowledge graph triples. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false, and show that existing calibration techniques are effective for KGE under this common but narrow assumption. Next, we introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained. Here, we show that existing calibration techniques are much less effective under the OWA than the CWA, and provide explanations for this discrepancy. Finally, to motivate the utility of calibration for KGE from a practitioner’s perspective, we conduct a unique case study of human-AI collaboration, showing that calibrated predictions can improve human performance in a knowledge graph completion task.

## introduction knowledge graphs are essential resources in natural language processing tasks such as question answering and reading comprehension @xcite . because they are by nature incomplete, extensive research efforts have been invested into completing them via different techniques @xcite . one such technique is knowledge graph embedding (kge), which involves learning latent representations of entities and relations to be used toward predicting new facts. kge models are most figure 1 : an example of how optimizing for ranking does not necessarily lead to trustworthy prediction scores. here, an uncalibrated kge model would perform well according to ranking metrics because the true triple is ranked highly, even though it receives a much lower score than the incorrect top-ranked triple and a similar score to the nonsensical triple below it. as such, the practical utility of kge for realworld knowledge graph completion remains limited, especially given that other completion techniques such as relation extraction and manual curation have already been reliably deployed in commercial and scientific settings @xcite @xcite . the outstanding issue that we address relates to trustworthiness: that is, to what degree can one trust the predictions made by kge? we believe that trustworthiness is an important part of making kge practical for knowledge graph completion. in this paper we propose to investigate confidence calibration as a technique toward making kge more trustworthy. intuitively, calibration is a post-processing step that adjusts kge link prediction scores to be representative of actual correctness probabilities @xcite . calibration has several benefits. from the systems perspective, natural language processing pipelines that include knowledge graphs can rely on calibrated confidence scores to determine which kge predictions to trust. from a practitioner's perspective, calibrated confidence scores act as decision support for accepting or verifying kge predictions. toward this direction we contribute the following: task we evaluate kge calibration for link prediction, which is important for making kge viable for deployment. while many knowledge graph embedding models exist, their calibration and general trustworthiness are under-explored ( § 2). ## related work while knowledge graph embeddings and calibration have both been extensively studied in separate communities-see @xcite for reviews of kge and @xcite for an overview of calibration for machine learningrelatively little work on calibration for knowledge graph embeddings exists. in the domain of relation extraction, a few works calibrate predicted knowledge graph triples as com-ponents of large-scale relation extraction systems. @xcite used platt scaling @xcite to calibrate the probabilities of factual triples in the proprietary knowledge vault dataset, and @xcite used platt scaling in a searchbased fact extraction system. however, we focus on link prediction with kge models that learn only from the knowledge graph itself ( § 3.1). we are aware of only two recent works that investigate calibration for kge, both of which address the task of triple classification @xcite . by contrast, we focus on link prediction, which is a different-and much more common (safavi and koutra, 2020)-kge evaluation task. we also contribute an evaluation under the open-world assumption, whereas tabacof and costabello (2020) evaluate under the closed-world assumption only. finally, unique to our work, we conduct a human-ai case study to demonstrate the benefits of calibration from a practitioner's perspective. ## closed-world evaluation we first evaluate kge calibration under the closedworld assumption (cwa), in which we assume triples not observed in a given knowledge graph are false. this assumption, which is standard in kge evaluation @xcite , helps narrow evaluation down to a well-defined task in which models are judged solely by their ability to fit known data. it is therefore important to first explore this (restrictive) assumption before moving to the more realistic but challenging owa ( § 5). ## open-world evaluation we now address the more realistic open-world assumption (owa), in which predictions not present in the knowledge graph are considered unknown, rather than false, until ground-truth labels are obtained. while the owa is beneficial because it helps us assess kge calibration under more realistic conditions, it is also challenging because it significantly changes the requirements for evaluation. specifically, now we need a label for every triple considered, whereas with the cwa we only needed labels for a small group of positives. we emphasize that this is the reason the owa is rarely used to evaluate kge. narrowing down the large space of unknowns to a manageable smaller set and labeling these triples can be difficult and costly. we thus contribute first steps toward evaluation strategies under the owa. ## case study finally, we conduct a case study of human-ai knowledge graph completion as a proof of concept on the benefits of kge calibration for practitioners. in this experiment, given "fill-in-the-blank" sentences corresponding to incomplete knowledge graph triples, the task is to choose from multiplechoice answer lists generated by kge to complete the sentences. we show that, compared to annotators not provided with confidence scores for this task, annotators provided with calibrated confidence scores for answer choices more accurately and efficiently complete triples. ## conclusion we investigate calibration as a technique for improving the trustworthiness of link prediction with kge, and uniquely contribute both closed-world and open-world evaluations; the latter is rarely studied for kge, even though it is more faithful to how practitioners would use kge for completion. we show that there is significant room for improvement in calibrating kge under the owa, and motivate the importance of this direction with our case study of human-ai knowledge graph completion. as knowledge graphs are increasingly used as gold standard data sources in artificial intelligence systems, our work is a first step toward making kge predictions more trustworthy. frank wilcoxon. 1992. individual comparisons by ranking methods. in breakthroughs in statistics, pages 196-202. springer. an yang, quan wang, jing liu, kai liu, yajuan lyu, hua wu, qiaoqiao she, and sujian li. 2019. enhancing pre-trained language representations with rich knowledge for machine reading comprehension. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2346-2357. bishan yang, wen-tau yih, xiaodong he, jianfeng gao, and li deng. 2015. embedding entities and relations for learning and inference in knowledge bases. in international conference on learning representations. bianca zadrozny and charles elkan. 2002. transforming classifier scores into accurate multiclass probability estimates. in proceedings of the eighth acm sigkdd international conference on knowledge discovery and data mining, pages 694-699. acm.