A Benchmark for Semi-Inductive Link Prediction in Knowledge Graphs (2023)

Abstract: Semi-inductive link prediction (LP) in knowledge graphs (KG) is the task of predicting facts for new, previously unseen entities based on context information. Although new entities can be integrated by retraining the model from scratch in principle, such an approach is infeasible for large-scale KGs, where retraining is expensive and new entities may arise frequently. In this paper, we propose and describe a large-scale benchmark to evaluate semi-inductive LP models. The benchmark is based on and extends Wikidata5M: It provides transductive, k-shot, and 0-shot LP tasks, each varying the available information from (i) only KG structure, to (ii) including textual mentions, and (iii) detailed descriptions of the entities. We report on a small study of recent approaches and found that semi-inductive LP performance is far from transductive performance on long-tail entities throughout all experiments. The benchmark provides a test bed for further research into integrating context and textual information in semi-inductive LP models.

## introduction a knowledge graph (kg) is a collection of facts describing relations between real-world entities. facts are represented in the form of subjectrelation-object triples such as (dave grohl, mem-berof, foo fighters). in this paper, we consider link prediction (lp) tasks, i.e., the problem of inferring missing facts in the kg. lp can be transductive (td; all entities known a priori), semi-inductive (si; some entities known a priori), and inductive (no entities known a priori). we concentrate on semi-inductive and transductive lp. si-lp focuses on modeling entities that are unknown or unseen during lp, such as out-of-kg entities (not part or not yet part of the kg) or newly created entities, e.g., a new user, product, or event. such previously unknown entities can be handled by retraining in principle. for large-scale kgs, however, retraining is inherently expensive and new entities may arise frequently. therefore, the goal of si-lp is to avoid retraining and perform lp directly, i.e., to generalize beyond the entities seen during training. to perform lp for unseen entities, context information about these entities is needed. the amount and form of context information varies widely and may take the form of facts and/or textual information, such as an entity mention and/or its description. for example, a new user in a social network may provide a name, basic facts such as gender or country of origin, and perhaps a textual selfdescription. in this paper, we introduce the wikidata5m-si benchmark for si-lp. our benchmark is based on the popular wikidata5m @xcite benchmark and has four major design goals: (g1) it ensures that unseen entities are long tail entities since popular entities (such as, say, foo fighters) and/or types and taxons (such as human and organization) are unlikely to be unseen. (g2) it allows to evaluate each model with varying amounts of contextual facts (0-shot, few-shot, transductive), i.e., to explore individual models across a range of tasks. (g3) it provides a controlled amount of textual information (none, mention, full description), where each setting demands different modeling capabilities. finally, (g4) the benchmark is large-scale so that retraining is not a suitable approach. all prior si-lp benchmarks violate at least one of these criteria. we report on a small experimental study with recent lp approaches. in general, we found that 1. si performance was far behind td performance in all experiments for long-tail entities, 2. there was generally a trade-off between td and si performance, 3. textual information was highly valuable, 4. proper integration of context and textual information needs further exploration, and 5. facts involving less common relations provided more useful context. our benchmark provides directions and a test bed for further research into si-lp. multiple si-lp datasets have been proposed in the literature. the benchmarks of daza et al. (2021), albooyeh et al. (2020), and galkin et al. (2021) are obtained by first merging the splits of smaller transductive lp datasets and subsequently sampling unseen entities uniformly to construct validation and test splits. these benchmarks do not satisfy goals g1-g4. shi and weninger (2018) follow a similar approach but focus on only 0-shot evaluation based on textual features. xie et al. (2016) and shah et al. (2019) select entities from freebase with connection to entities in fb15k @xcite , also focussing on 0-shot evaluation using rich textual descriptions. these approaches do not satisfy g2 and g3. finally, @xcite and @xcite uniformly sample test triples and mark occurring entities as unseen. these approaches do not focus on long-tail entities (and, in fact, the accumulated context of unseen entities may be larger than the training graph itself) and they do not satisfy g1-g3. there are also several of fully-inductive lp benchmarks @xcite involving kgs. while si-lp aims to connect unseen entities to an existing kg, fully-inductive lp reasons about a new kg with completely separate entities (but shared relations). we do not consider this task in this work. ## the wikidata5m-si benchmark wikidata5m-si is based on the popular wiki-data5m @xcite benchmark, which is induced by the 5m most common entities of wikidata. our benchmark contains transductive and semi-inductive valid/test splits; see tab. 1 for an overview. generally, we aimed to keep wikidata5m-si as close as possible to wikidata5m. we did need to modify the original transductive valid and test splits, however, because they unintentionally contained both seen and unseen entities (i.e., these splits were not fully transductive). we did that by simply removing all triples involving unseen entities. unseen entities. to ensure that unseen entities in the semi-inductive splits are from the long tail (g1), we only considered entities of degree 20 or less. to be able to provide sufficient context for few-shot tasks (g2), we further did not consider entities of degree 10 or less. in more detail, we sampled 500 entities of degrees 11-20 (stratified sampling grouped by degree) for each semi-inductive split. all sampled entities, along with their facts, were removed from the train split. note that these entities (naturally) have a different class distribution than all entities; see sec. a.1 for details. tasks and metrics. for td tasks, we follow the standard protocol of wikidata5m. to construct si tasks, we include 11 of the original facts of each unseen entity into its si split; each split thus contains 5,500 triples. this enables up to 10-shot si tasks (1 fact to test, up to 10 facts for context). for entities of degree larger than 11, we select the 11 facts with the most frequent relations; see tab. 2 for an example. the rationale is that more common relations (such as instanceof or country) may be considered more likely to be provided for unseen entities than rare ones (such as militarybranch or publisher). we then construct a single k-shot task for each triple (s, p, o) in the si split as follows. when, say, s is the unseen entity, we consider the lp task (s, p, ?) and provide k additional facts of form (s, p , o ) as context. context facts are selected by frequency as above, but we also explored random and infrequent-relation context in our study. models are asked to provide a ranking of predicted answers, and we determine the filtered mean reciprocal rank (mrr) and hits@k of the correct answer (o). textual information. for each entity, we provide its principal mention and a detailed description (both directly from wikidata5m); see tab. ## semi-inductive link prediction models we briefly summarize recent models for si-lp; we considered these models in our experimental study. graph-only models. complex @xcite is the best-performing transductive kge model on wikidata5m @xcite . to use complex for si-lp, we follow an approach explored by @xcite . in particular, we represent each entity as the sum of a local embedding (one per entity) and a global bias embedding. for 0-shot, we solely use the global bias for the unseen entity. for k-shot, we obtain the local embedding for the unseen entity by performing a single training step on the context triples (keeping all other embeddings fixed). an alternative approach is taken by odistmult-eravg @xcite , which represents unseen entities by aggregating the embeddings of the relations and entities in the context. foot_0 a more direct approach is taken by hitter @xcite , which contextualizes the query entity with its neighborhood for td-lp. the approach can be used for si-lp directly by using a masking token (akin to the global bias above) for an unseen entity. we originally planned to consider nodepiece @xcite (entity represented by a combination of anchor embeddings) and nbfnet @xcite (a gnn-based lp model); both support si-lp directly. however, the available implementations did not scale to wikidata5m-si (out of memory). 2 text-based models. as a baseline approach to integrate textual information directly into kge models, we consider the approach explored in the wikikg90m benchmark @xcite ; see sec. a.2 for details. the remaining approaches are purely textual. simkgc @xcite utilizes two pretrained bert transformers: one to embed query entities (and relations) based on their mention or description, and one for tail entities. using a contrastive learning approach, it measures cosine similarity between both representations for ranking. kgt5 @xcite is a sequence-to-sequence link prediction approach, which is trained to generate the mention of the answer entity using the mention or description of the query entity and relation as input. both approaches support 0-shot si-lp when textual information is provided for the query entity. they do not utilize additional context, however, i.e., do not support k-shot si-lp. kgt5-context @xcite is an extension of kgt5, which extends the input of kgt5 by the one-hop neighborhood of the query entity and consequently supports k-shot lp directly. ## experimental study we evaluated all presented baseline models in the td and si setting on the atomic, mentions, and descriptions dataset. further, we evaluated in detail which context was most useful and what information was conveyed by textual mentions and descriptions. setup. source code, configuration, and the benchmark itself are available at https://github. com/uma-pi1/wikidata5m-si . for further details on hyperparameter tuning and training see sec. a.3. ## conclusion we proposed the new wikidata5m-si large-scale benchmark for semi-supervised link prediction. the benchmark focuses on unseen entities from the long tail and allows to evaluate models with varying and controlled amounts of factual and textual context information. in our experimental evaluation, we found that semi-inductive lp performance fell behind transductive performance for long-tail entities in general, and that detailed textual information was often more valuable than factual context information. moreover, current models did not integrate these two types of information adequately, suggesting a direction for future research.