KG x B oard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models (2022)

Abstract: Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. To augment KGs with new knowledge, researchers proposed models for KG Completion (KGC) tasks such as link prediction; i.e., answering (h; p; ?) or (?; p; t) queries. Such models are usually evaluated with averaged metrics on a held-out test set. While useful for tracking progress, averaged single-score metrics cannotreveal what exactly a model has learned — or failed to learn. To address this issue, we propose KGxBoard: an interactive framework for performing fine-grained evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a KGC model. In our experiments, we highlight the findings that we discovered with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.

## introduction knowledge graphs (kgs) are graph databases that store information about entities and the relations between them in the form of (head, predicate, tail)triples @xcite . because of their flexible structure, kgs are used for storing general real-world data @xcite as well as domain-specific data covering various domains (abu-salih, 2021), including medicine @xcite , iot @xcite and finance @xcite . kgs play an important role in nlp: they are used for many downstream tasks, including language modeling @xcite , entity linking @xcite and question answering @xcite . a common problem for kgs is that they are incomplete (i.e., they do not contain all facts about the world or the particular domain), which could lead to limitations in performance for the downstream tasks. to tackle this problem of incompleteness, the research community has worked on many kg completion (kgc) tasks, most prominently on link prediction: predicting new facts within the kg by providing ranked predictions to the queries (h; p; ?) or (?; p; t) @xcite . models for kgc tasks are typically evaluated with singlescore metrics that are averaged over a held-out test set. for instance, for a kgc query-e.g., (h; p; ?)-, hits@k indicates the average number of correct answers from the test set that appear within the top-k ranked entities predicted by the kgc model. 2 while using such scores is important for tracking the progress of kgc models, researchers observed that a more fine-grained evaluation is needed @xcite , because the averaged metrics cannot answer the question of what properties have the models actually learned-or failed to learn. to investigate what properties were learned by the kgc models, researchers have designed specific datasets and experimental setups. for example, @xcite used the idea of behavioral testing applied to nlp models @xcite to perform more fine-grained tests for relation symmetry. in particular, they measured the performance of kgc models for queries that contain symmetric relations; i.e., relations that are true for both (h; p; t) and (t; p; h) such as (x; marriedto; y). however, their proposed framework only contains a limited number of tests and might not cover model properties of interest to other researchers. to make the fine-grained evaluation more generic-and to compare different kgc models across different properties-we propose kgxboard: a method and software implementation for fine-grained evaluation of kgc models (see illustration in fig. 1 ). kgxboard splits the evaluated data into groups (buckets) according to certain properties of the data. for instance, one can , partitions of the evaluated data). for example, we observed that the tucker model performs very well on triples with manyto-one (m-1) relations. however, it performs poorly on triples with one-to-many (1-m) relations and symmetric relations. with the multiple analysis mode, the user can compare the overall and bucketized performance across different models. such a view enables the user to compare the models in a fine-grained manner; e.g., rescal is ranked 4th in the overall performance, but it is ranked 1st for the triples having symmetric relations. likewise, distmult is ranked 5th in the overall performance and 1st for the triples with one-to-one relations. in both modes, the user can interact with the ui to see details of the data buckets, change the evaluation metrics, etc. kgxboard builds upon explainaboard @xcite , which is an explainable leaderboard for nlp tasks. we adapt explainaboard to the kgc tasks 3 by providing: (1) a method and software implementation for fine-grained evaluation of kgc models, integrated into explainaboard; ## preliminaries the performance of kgc models is evaluated on a held-out test set of golden kg triples. in particular, the models return a set of ranked answers to the queries-e.g., (h; p; ?)-, where the correctness of the answers is evaluated against the golden triples. the final score is averaged over the examples from the held-out test set. the standard scores used in the literature are: ## kgxboard's fine-grained evaluation prior work in nlp has pointed to the issues of using single-score metrics, which do not expose what exact properties of the data were (or were not) learned by the models @xcite ; e.g., some information extraction models perform poorly when there is a conjunction present in the "custom_features": { "rel_type": { "dtype": "string", "description": "predicate symmetry", "num_buckets": 2 } } figure 2 : defining custom features for bucketization of the evaluation data (e.g., validation or test data). following explainaboard @xcite , the basic idea of kgxboard's fine-grained evaluation is to breakdown the performance measure (e.g., hits@10) over individual groups (buckets) in addition to the performance score over the overall evaluation dataset. this approach involves three steps: (i) define features upon which the evaluation dataset is going to be partitioned; (ii) partition evaluation dataset into different buckets based on the defined features; and (iii) calculate performance w.r.t. each bucket. in contrast to explainaboard, kgxboard is tailored for the kg completion tasks and their evaluation metrics. ## kgxboard: system overview a general overview of the kgxboard architecture is illustrated in figure 3 . the users can provide the data for the models through the front-end (via the ui) or the back-end (via the cli or the sdk). then, these results are stored in a database (db), which can be accessed and viewed either with the visual interface, programmatically, or with the commandline interface. in general, the users can choose three ways to use kgxboard's functionalities: (i) directly from the interactive web interface; (ii) through an api from kgc frameworks (pykeen and libkge); (iii) through the command-line interface with already provided data. ## conclusions we presented kgxboard: an interactive framework for fine-grained and interpretable evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a kgc model. we highlighted insights that would be impossible to detect with standard leaderboards. recent work in nlp has pointed to the problems of using single-score metrics @xcite @xcite . in particular, such single-score metric evaluations do not expose the data-centric properties that were not learned by the models (e.g., problems with learning named entities that span across multiple tokens). in nlp, such issues are tackled by proposing multifaceted and/or explainable leaderboards and benchmarks @xcite @xcite @xcite @xcite . to the best of our knowledge, there is no prior work that proposes such multi-faceted leaderboards for kg completion models. a.2 evaluation of kgc models kgc benchmarks. prior work has shown that learning latent representations-i.e., kg embeddings (kges)-for the entities and relations is highly beneficial for tackling kgc tasks @xcite . researchers have evaluates the effects of hyperparameters (and different training strategies) of kge models for link prediction. in particular, @xcite studied the effects of different training objectives on the distmult model @xcite and found that using the cross entropy loss function is a better alternative to the binary cross entropy. @xcite performed a systematic study on many kge models with the libkge framework @xcite and found that training components make a huge difference in model performance. @xcite proposed a more data-centric benchmark (codex), which improves upon previous benchmarks by proposing additional datasets extracted from wikidata @xcite and then using them to improve current kge models. @xcite proposed an evaluation framework such that it breaks ties of same-score answers according to different strategies. pykeen @xcite is another kge framework and benchmark, which facilitates the training and evaluation of kge models for kg completion tasks across a variety of datasets. although highly useful for overall evaluation, none of these frameworks provide fine-grained evaluation of kge models, nor an interactive interface for such evaluation.