Exploring the Limits of Few-Shot Link Prediction in Knowledge Graphs (2021)

Abstract: Real-world knowledge graphs are often characterized by low-frequency relations—a challenge that has prompted an increasing interest in few-shot link prediction methods. These methods perform link prediction for a set of new relations, unseen during training, given only a few example facts of each relation at test time. In this work, we perform a systematic study on a spectrum of models derived by generalizing the current state of the art for few-shot link prediction, with the goal of probing the limits of learning in this few-shot setting. We find that a simple, zero-shot baseline — which ignores any relation-specific information — achieves surprisingly strong performance. Moreover, experiments on carefully crafted synthetic datasets show that having only a few examples of a relation fundamentally limits models from using fine-grained structural information and only allows for exploiting the coarse-grained positional information of entities. Together, our findings challenge the implicit assumptions and inductive biases of prior work and highlight new directions for research in this area.

## introduction a knowledge graph (kg) is a multi-relational graph that offers a structured way to organize facts about the world. encoder-decoder approaches are commonly used to predict new facts from existing ones where entities and relations are embedded in a lowdimensional vector space via an encoder, to then score the likelihood of observing a new fact via a decoder @xcite @xcite . it is well known that the performance of these methods can significantly drop when predicting for relations that are only observed in a few example facts. however, link prediction for these low-frequency relations is very important, as not only are these relations abundant in most knowledge graphs, they are also key for knowledge graph to study this low-frequency regime, @xcite created the nell-one and wiki-one benchmarks where the task is to predict new facts for a set of new relations at test time, where each relation is only observed a few times (as specified by some small fixed number k). previous approaches have shown promising results using metric-based @xcite and gradient based meta-learning techniques @xcite . however, we argue that these models are limited by the current task formulation to only exploit coarse-grained positional signals (i.e., nodes belonging to the same community) that are abundant in these benchmarks, rather than leveraging structural signals (e.g. transitivity, symmetry). present work. in this work, we take a critical take on current approaches for few-shot link prediction over knowledge graphs. we posit that current metalearning based approaches benefit largely due to the positional signals in entities, rather than utilising information about the low frequency relations. we corroborate these insights by conducting a systematic study on a spectrum of models with decreasing complexity. interestingly, we find that a much simpler zero-shot variant of the state of the art -devoid of any meta-learning schemeyields surprisingly competitive results, while not consuming any example facts about a relation. motivated by these observations, we design a set of null models tailored to different learning signals a model might utilize to drive effective link predic-tion. empirically, we validate that these existing meta-learning models are ill-equipped to infer logical patterns about the few-shot relations. these findings bring forth the shortcomings of the current task formulation and raises new questions in both task and model design while highlighting new directions of research in few-shot link prediction. ## null models in order to probe and understand the performance of different models, we introduce two null models, which are used to generate synthetic data that satisfy certain properties. motivated by recent literature on position versus structure-aware methods in relational learning @xcite , we test the models' ability to learn from two key sources of information: structural information and positional information. in the context of knowledge graphs, structural information corresponds to the fine-grained relational semantics. these are the logical patterns that are extracted by state-of-the-art rule induction systems, such as rulen @xcite . on the other hand, positional information corresponds to the coarse-grained community structure of the nodes in the graph. in other words, two nodes are said to be positionally 'close' in the graph, i.e., if they belong to the same community @xcite . ## experiments we followed the same experimental setup as in @xcite , as described in appendix a. we conducted our experiments on the nell-one and wiki-one datasets 1 . for more details on these benchmarks, we refer the reader to ## conclusion we conducted a systematic study of various models to probe their limits in performing few-shot link prediction. our experiments on both synthetic and real data show that the current task formulation encourages models to mainly rely on positional information around entities, rather than leveraging logical signals about relations. in fact, we empirically show that having only k examples of a relation fundamentally limits the types of logical patterns that can be learned. we argue that a future direction in few-shot link prediction should allow for a more careful construction of the support set, to scaffold the use of logical patterns in few-shot learning.