Eureka: Neural Insight Learning for Knowledge Graph Reasoning (2022)

Abstract: The human recognition system has presented the remarkable ability to effortlessly learn novel knowledge from only a few trigger events based on prior knowledge, which is called insight learning. Mimicking such behavior on Knowledge Graph Reasoning (KGR) is an interesting and challenging research problem with many practical applications. Simultaneously, existing works, such as knowledge embedding and few-shot learning models, have been limited to conducting KGR in either “seen-to-seen” or “unseen-to-unseen” scenarios. To this end, we propose a neural insight learning framework named Eureka to bridge the “seen” to “unseen” gap. Eureka is empowered to learn the seen relations with sufficient training triples while providing the flexibility of learning unseen relations given only one trigger without sacrificing its performance on seen relations. Eureka meets our expectation of the model to acquire seen and unseen relations at no extra cost, and eliminate the need to retrain when encountering emerging unseen relations. Experimental results on two real-world datasets demonstrate that the proposed framework also outperforms various state-of-the-art baselines on datasets of both seen and unseen relations.

## introduction human knowledge provides a formal understanding of the world. knowledge graphs (kgs) that represent structural relations between entities in the form of (head entity, relation, tail entity) have become an increasingly popular research direction towards cognition and human-level intelligence. these triples of kgs are abbreviated using (h, r, t) in this paper. over the last few years, the works @xcite on knowledge embedding have achieved impressive results in the knowledge graph reasoning (kgr) task. to * corresponding author successfully learn a set of relations, these methods usually require a large number of training triples and cannot infer missing facts of unseen relations due to the sparse interactions, which are essentially transductive learning processes in terms of the relations. thus, we categorize this line of knowledge embedding research as "seen-to-seen" methods; i.e., reasoning from seen relations to seen relations. moreover, the representations of the relations in kgs produced by knowledge embedding models always remain fixed after training. they may be sub-optimal since the real-world large-scale kgs dynamically evolve quickly with new relations emerging every day, rather than staying static. suppose that we would like to expand the set of relations that the knowledge embedding models can recognize. we need to collect training triples for the emerging (unseen) relations; i.e., those not in the initial training set, and then restart the aforementioned computationally costly training procedure on the enhanced training set. not to mention the fact that the model may not perform well when only few training examples are available for the unseen relations @xcite . to alleviate the above challenge, some few-shot learning methods @xcite have been proposed, which can be seen as inductive learning approaches. their basic ideas are to predict new facts in a meta-learning framework in a setting where only few training triples for each unseen relation are available. we term them as "unseen-to-unseen" methods; i.e., reasoning from unseen relations to unseen relations. this is possible since the meta-learning framework can simulate the unseen relations during meta-training, while they are unobservable in conventional learning schemes. however, their performance will reach a plateau as the number of training examples increases, as illustrated in figure 1 . moreover, they cannot perform as well as knowledge embedding models on the initial seen relations with sufficient training triples, as the few-shot learners are adapted to the common parts of the different metatasks and forget the relation-specific information. in other words, the dramatic performance of fewshot learning methods on unseen relations comes at the cost of dysfunction on seen relations with sufficient training examples available. it is also different from the human learning system, where new concepts can be learned from very few examples at no extra cost. meanwhile, the human learning system exhibits the remarkable ability to effortlessly discover novel concepts during the "eureka moment", with only one or few examples as the trigger. for example, a child, having accumulated enough knowledge (seen relations) like "ceo", "president" and so on, can easily learn and generalize the unseen concept of "leader"from only a single knowledge set like (gandhi, leader_of, india) by analogy. mimicking this behavior in artificial reasoning systems is an interesting and very challenging research problem with many practical advantages, such as developing real-time knowledge reasoning systems for downstream applications, such as language models. motivated by the limitations of knowledge embedding methods and few-shot kgr models, we mimic human insight learning modes in machine learning and propose a neural insight learning framework termed eureka for kgr tasks. the whole structure of eureka is illustrated in figure 2 . eureka aims to tackle the problem knowledge embedding methods and few-shot kgr models encountered, under a more realistic setting, where a large set of training triples are assumed to exist for seen relations; and using these data as the sole input, we want to develop a kgr model that, is not only capable of recognizing these seen relations, but also learning unseen relations from only one training example (provided only at the testing time) without sacrificing the performance on seen relations or requiring to be retrained). we also devise a cross-domain attention (cda) network to model the semantic interactions and bridge the gap between unseen and seen relations; for example, the unseen relation "leaderof" has semantic similarity with the seen relations "presidentof" and "ceoof" and modeling such similarity will help to make up for the lack of training information of unseen relations and represent the unseen relation more accurately to some extent. compared to prior approaches, we believe that eureka resembles more closely the human learning behavior (w.r.t. how it learns novel concepts). eureka is also more suitable in the real-world scenario where unseen relations do not emerge one by one but may emerge simultaneously as a set, with only few triples available for each new unseen relation. to summarize, the contributions are as follows: ## related work the neural insight learning framework draws on the previous research in knowledge embedding and meta-learning. ## preliminaries in this section, we formally describe neural insight learning in the kgr scenario and leave technical details to the next section. according to the gestalt theory of learning, insight learning occurs spontaneously when people discover new knowledge within their prior knowledge as a result of reasoning or problem-solving processes that reorganize or restructure that knowledge @xcite . in other words, there are two key points about machine insight learning in the kgr scenario. one is that the model could learn new unseen relations with a trigger based on the prior seen relations it learned, the other is that the model should achieve good performance on both seen and unseen relations as human beings do. we first present the definition of kgr, then formalize neural insight learning in the kgr scenario. the difference between eureka and previous relevant learning theories such as meta-learning @xcite , transfer learning (pan and yang, 2010), one-shot learning @xcite , and one-pass learning @xcite are also discussed. ## neural insight learning 4.1 overall architecture to fully mimic human insight learning behaviors, eureka is built on a two-stage learning framework. figure 2 shows an overview of eureka. in the prior knowledge learning stage, eureka takes triples sampled from the g a as input, similar to the knowledge embedding models @xcite based on deep neural networks. we use a bidirectional transformer encoder and a cnn-based decoder to learn the dynamic embeddings of entities and relations of g a and the parameters of the encoderdecoder model. the second stage, termed trigger learning, is designed to learn new knowledge with only one training example as a trigger based on the prior knowledge eureka gained. thus, we adopt a meta-learning framework to implement trigger learning. during meta-training step, the trigger sampled from g b only contains one training triple (h 0 , r , t 0 ) for each unseen relation r . the representations of r can be produced by the trained encoder-decoder model. however, only a single training example cannot guarantee the accurate representations of r as the previous research on knowledge embedding @xcite claimed. we use a cda network to make up for the lack of training information of unseen relations. the cda mechanism incorporates the embeddings of the relevant seen relations with the calibrated embeddings of r to acquire accurate representations of unseen relations. the same cnn-based decoder is then applied to judge whether the query is true or not with the given trigger during the meta-testing step. ## experiments we investigate three issues with eureka: (1) could eureka improve the performance of kgr on unseen relations at no extra cost? (2) is each component in eureka necessary? (3) how cda works? to answer these questions, we conduct experiments on two kg datasets and systematically analyze the corresponding results. ## conclusion in this work, we present a neural insight learning framework (eureka), which mimics human insight learning modes to bridge the "seen" to "unseen" gap in the kgr tasks. we train eureka in prior knowledge learning and trigger learning stages. specifically, eureka acquires the representations of entities and seen relations in the prior knowledge learning stage, and then learns the unseen relations efficiently through a cda network with the incorporation of the embeddings of seen relations. eureka meets our expectation of the model to not only have good performance on both relation types but also eliminate the need to retrain the original training datasets. the experimental results demonstrate that our model outperforms the state-of-the-art baselines on datasets of both seen and unseen relations. the case studies confirm the cda network is empowered to select relevant seen relations to better represent unseen relations. we plan to investigate enhancing cda with relations' text descriptions as future directions of this work.