ACTC : Active Threshold Calibration for Cold-Start Knowledge Graph Completion (2023)

Abstract: Self-supervised knowledge-graph completion (KGC) relies on estimating a scoring model over (entity, relation, entity)-tuples, for example, by embedding an initial knowledge graph. Prediction quality can be improved by calibrating the scoring model, typically by adjusting the prediction thresholds using manually annotated examples. In this paper, we attempt for the first time cold-start calibration for KGC, where no annotated examples exist initially for calibration, and only a limited number of tuples can be selected for annotation. Our new method ACTC finds good per-relation thresholds efficiently based on a limited set of annotated tuples. Additionally to a few annotated tuples, ACTC also leverages unlabeled tuples by estimating their correctness with Logistic Regression or Gaussian Process classifiers. We also experiment with different methods for selecting candidate tuples for annotation: density-based and random selection. Experiments with five scoring models and an oracle annotator show an improvement of 7% points when using ACTC in the challenging setting with an annotation budget of only 10 tuples, and an average improvement of 4% points over different budgets.

## introduction knowledge graphs (kg) organize knowledge about the world as a graph where entities (nodes) are connected by different relations (edges). the knowledge-graph completion (kgc) task aims at adding new information in the form of @xcite triples to the knowledge graph. the main objective is assigning to each triple a plausibility score, which defines how likely this triple belongs to the underlying knowledge base. these scores are usually predicted by the knowledge graph embedding (kge) models. however, most kgc approaches do not make any binary decision and provide a ranking, not classification, which does not allow one to use them as-is to populate the kgs @xcite . to transform the scores into predictions (i.e., how probable is it that this triple should be included in the kg), decision thresholds need to be estimated. then, all triples with a plausibility score above the threshold are classified as positive and included in the kg; the others are predicted to be negatives and not added to the kg. since the initial kg includes only positive samples and thus cannot be used for threshold calibration, the calibration is usually performed on a manually annotated set of positive and negative tuples (decision set). however, manual annotation is costly and limited, and, as most knowledge bases include dozens @xcite , hundreds @xcite or even thousands @xcite of different relation types, obtaining a sufficient amount of labeled samples for each relation may be challenging. this raises a question: how to efficiently solve the cold-start thresholds calibration problem with minimal human input? we propose a new method for active threshold calibration actc foot_0 , which estimates the relation thresholds by leveraging unlabeled data additionally to human-annotated data. in contrast to already existing methods @xcite ) that use only the annotated samples, actc labels additional samples automatically with a trained predictor (logistic regression or gaussian process model) estimated on the kge model scores and available annotations. a graphical illustration of actc is provided in figure 1 . our main contributions are: experiments on several datasets and with different kge models demonstrate the efficiency of actc for different amounts of available annotated samples, even for as little as one. ## related work knowledge graph embedding methods @xcite @xcite have been originally evaluated on ranking metrics, not on the actual task of triple classification, which would be necessary for kgc. more recent works have acknowledged this problem by creating data sets for evaluating kgc (instead of ranking) and proposed simple algorithms for finding prediction thresholds from annotated triples @xcite . in our work, we study the setting where only a limited amount of such annotations can be provided, experiment with different selection strategies of samples for annotation, and analyze how to use them best. @xcite have studied active learning for selecting triples for training a scoring model for kg triples, but their method cannot perform the crucial step of calibration. they consequently only evaluate on ranking metrics, not measuring actual link prediction quality. in contrast, our approach focuses on selecting much fewer samples for optimal calibration of a scoring model (using positive, negative, and unlabeled samples). ## actc: active threshold calibration actc consists of three parts: selection of samples for manual annotation, automatic labeling of additional samples, and estimating the per-relation thresholds based on all available labels (manual and automatic ones). the first step is selecting unlabeled samples for human annotation. in actc this can be done in two ways. one option is a random sampling from the set of all candidate tuples (actc rndm ; the pseudocodes can be found in algorithm 1). however, not all annotations are equally helpful and informative for estimation. to select the representative and informative samples that the system can profit the most from, especially with a small annotation budget, we also introduce density-based selection actc dens inspired by the density-based selective sampling method in active learning @xcite (the pseudocode can be found in algorithm 2 in appendix a). the sample density is measured by summing the squared distances between this sample's score (predicted by the kge model) and the scores of other samples in the unlabeled dataset. the samples with the highest density are selected for human annotation. in a constrained-budget setting with a limited amount of manual annotations available, there are sometimes only a few samples annotated for some relations and not even one for others. to mitigate this negative effect and to obtain good thresholds even with limited manual supervision, actc labels more samples (in addition to the manual annotations) with a classifier trained on the manually annotated samples to predict the labels based on if @xmath0: train a classifier cr on scores gold r and y gold r 11: t.append(τ ) the kge model scores. we experiment with two classifiers: logistic regression (actc-lr) and gaussian processes (actc-gp). the amount of automatically labeled samples depends on hyperparameter n, which reflects the minimal amount of samples needed for estimating each threshold (see ablation study of different n values in section 5). if the number of samples annotated for a relation r (l r ) is larger or equal to n, only these l r annotated samples are used for threshold estimation. if the amount of manually annotated samples is insufficient (i.e., less than n), the additional nl r samples are randomly selected from the dataset and labeled by a lr or gp classifier. the automatically labeled and manually annotated samples build a per-relation threshold decision set, which contains at least n samples for a relation r with (manual or predicted) labels. the threshold for relation r is later optimized on this decision set. the final part of the algorithm is the estimation of the relation-specific thresholds. each sample score from the decision set is tried out as a potential threshold; the relation-specific thresholds that maximize the local accuracy (calculated for this decision set) are selected. ## experiments we evaluate our method on two kgc benchmark datasets extracted from wikidata and augmented with manually verified negative samples: codex-s and codex-m foot_1 @xcite . some details on their organization are provided in appendix b. the kge models are trained on the training sets foot_2 . the actc algorithm is applied on the validation sets: the gold validation labels are taken as an oracle (manual annotations; in an interactive setting they would be presented to human annotators on-the-fly); the remaining samples are used unlabeled. the test set is not exploited during actc training and serves solely for testing purposes. the dataset statistics are provided in ## ablation study a more detailed ablation study of different actc settings is provided in appendix d. ## conclusion in this work, we explored for the first time the problem of cold-start calibration of scoring models for knowledge graph completion. our new method for active threshold calibration actc provides different strategies of selecting the samples for manual annotation and automatically labels additional tuples with logistic regression and gaussian processes classifiers trained on the manually annotated data. experiments on datasets with oracle positive and negative triple annotations, and several kge models, demonstrate the efficiency of our method and the considerable increase in the classification performance even for tiny annotation budgets.