A daptive A ttentional N etwork for F ew- S hot K nowledge G raph C ompletion (2020)

Abstract: Few-shot Knowledge Graph (KG) completion is a focus of current research, where each task aims at querying unseen facts of a relation given its few-shot reference entity pairs. Recent attempts solve this problem by learning static representations of entities and references, ignoring their dynamic properties, i.e., entities may exhibit diverse roles within task relations, and references may make different contributions to queries. This work proposes an adaptive attentional network for few-shot KG completion by learning adaptive entity and reference representations. Specifically, entities are modeled by an adaptive neighbor encoder to discern their task-oriented roles, while references are modeled by an adaptive query-aware aggregator to differentiate their contributions. Through the attention mechanism, both entities and references can capture their fine-grained semantic meanings, and thus render more expressive representations. This will be more predictive for knowledge acquisition in the few-shot scenario. Evaluation in link prediction on two public datasets shows that our approach achieves new state-of-the-art results with different few-shot sizes. The source code is available at https://github.com/JiaweiSheng/FAAN .

## introduction knowledge graphs (kgs) like freebase @xcite , nell @xcite and wikidata @xcite are extremely useful resources for nlp tasks, such as information extraction @xcite , machine reading @xcite , and relation extraction @xcite . a typical kg is a multi-relational graph, represented as triples of the form (h, r, t), indicating that two entities are connected by relation r. although a kg contains a great number of triples, it is also known to suffer from incompleteness problem. kg completion, which aims at automatically inferring missing facts by examining existing ones, has thus attracted broad attention. a promising approach, namely kg embedding, has been proposed and successfully applied to this task. the key idea is to embed kg components, including entities and relations, into a continuous vector space and make predictions with their embeddings. current kg embedding methods mostly require sufficient training triples for all relations to learn expressive representations (i.e., embeddings). in real kgs, a large portion of kg relations is actually long-tail, having only a limited (few-shot) number of relational triples @xcite . this may lead to low performance of embedding models on kg completion for those long-tail relations. recently, several studies @xcite @xcite have pro-posed to address the few-shot issue of kg completion, where one task is to predict tail entity t in a query (h, r, ?) given only a few entity pairs of the task relation r. these known few-shot entity pairs associated with r are called references. to improve semantical representations of the references, @xcite and @xcite devise modules to enhance entity embeddings with their local graph neighbors. the former simply assumes that all neighbors contribute equally to the entity embedding, and in this way the neighbors are always weighted identically. the latter develops the idea by employing an attention mechanism to assign different weights to neighbors, but the weights do not change throughout all task relations. therefore, both works assign static weights to neighbors, leading to static entity representations when involved in different task relations. we argue that entity neighbors could have varied impacts associated with different task relations. figure 1(a) gives an example of head entity billgates associated with two task relations. the left neighbors show his business role, while the right ones show his family role, which reveals quite different meanings. intuitively, the task relation ceoof is supposed to pay more attention to the business role of entity billgates than the other one. in addition, task relations can be polysemous, also showing different meanings when involved in different entity pairs. therefore, the reference triples could also make different contributions to a particular query. take a task relation subpartof as an example. as shown in figure 1 (b), subpartof associates with different meanings, e.g., organization-related as (cavaliers, subpartof, nba) and location-related as (petersburg, subpartof, virginia). obviously, for query (chicagobulls, subpartof, ?), referring to the organization-related references would be more beneficial. to address the above issues, we propose an adaptive attentional network for few-shot kg completion (faan), a novel paradigm that takes dynamic properties into account for both entities and references. specifically, given a task relation with its reference/query triples, faan proposes an adaptive attentional neighbor encoder to model entity representations with one-hop entity neighbors. unlike the previous neighbor encoder with a fixed attention map in @xcite , we allow attention scores dynamically adaptive to the task relation under the translation assumption. this will capture the diverse roles of entities through varied impacts of neighbors. given the enhanced entity representations, faan further adopts a stack of transformer blocks for reference/query triples to capture multi-meanings of the task relation. then, faan obtains a general reference representation by adaptively aggregating the references, further differentiating their contributions to different queries. as such, both entities and references can capture their fine-grained meanings, and render richer representations to be more predictive for knowledge acquisition in the few-shot scenario. the contributions of this paper are three-fold: ## related work recent years have seen increasing interest in learning representations for entities and relations in kgs, a.k.a kg embedding. various methods have been devised, and roughly fall into three groups: 1) translation-based models which interpret relations as translating operations between head-tail entity pairs @xcite ; 2) simple semantic matching models which compute composite representations over entities and relations using linear mapping operations @xcite @xcite ; and 3) (deep) neural network models which obtain composite representations using more complex operations @xcite . please refer to @xcite @xcite for a thorough review of kg embedding techniques. traditional embedding models always require sufficient training triples for all relations, thus are limited when solving the few-shot problem. previous few-shot learning studies mainly focus on computer vision @xcite , imitation learning @xcite and sentiment analysis @xcite . recent attempts @xcite @xcite tried to perform few-shot relational learning for long-tail relations. @xcite proposed a matching network gmatching, which is the first research on one-shot learning for kgs as far as we know. gmatching exploits a neighbor encoder to enhance entity embeddings from their one-hop neighbors, and uses a lstm matching processor to perform a multi-step matching by a lstm block. fsrl @xcite extends gmatching to few-shot cases, further capturing local graph structures with an attention mechanism. @xcite proposed a novel meta relational learning framework metar by extracting and transferring shared knowledge across tasks from a few existing facts to incomplete ones. however, previous studies learn static representations of entities or references, ignoring their dynamic properties. this work attempts to learn dynamic entity and reference representations by an adaptive attentional network. dynamic properties have also been explored in other contexts outside few-shot relational learning. @xcite ; @xcite performed kg completion by learning dynamic entity and relation representations, but their methods are specially devised for traditional kg completion. @xcite adopted an adaptive attentional model for image captioning. @xcite tried to model dynamic user preference using a recurrent network with adaptive attention for the sequential recommendation. all these studies demonstrate the capability of modeling dynamic properties to enhance learning algorithms. ## background gonsider a kg g containing a set of triples @xmath1, r, t) ∈ e × r × e}, where e and r denotes the entity set and relation set, respectively. this work focuses on a challenging link prediction scenario, i.e., few-shot kg completion. we follow the standard definition of this task @xcite : given a relation @xmath2, t k )|(h k , r, t k ) ∈ t }, one task is to complete triple (h, r, t) with tail entity @xmath3, i.e., to predict t from a candidate entity set c given (h, r, ?). when |s r | = k and k is very small, the task is called k-shot kg completion. for this task, the goal of a few-shot learning method is to rank the true tail entity higher than false candidate entities, given few-shot reference entity pairs s r . to imitate such a link prediction task, each training task corresponds to a relation @xmath4, i.e., d @xmath5, q r }, where s r only consists of k-shot reference entity pairs (h k , t k ). additionally, q @xmath6, t m /c hm,r )} contains all queries with ground-truth tail entity t m and the corresponding candidates c hm,r , where each candidate is an entity in e selected based on the entity type constraint @xcite . the few-shot learning method thus could be trained on the task set by ranking the candidates in c hm,r given the query (h m , r, ?) and its references s r . all tasks in training form the meta-training set, denoted as t @xmath7. here, we only consider a closed set of entities appearing in e. after suffcient training with meta-training set, the learned model can be used to predict facts of new relation @xmath8. the relations used for testing are unseen from meta-training, i.e., r ∪ @xmath9. each testing relation r also has its own few-shot references and queries, i.e., d @xmath10, q r }, defined in the same way as in meta-training. all tasks in testing form the metatesting set, denoted as t @xmath11. in addition, we also suppose that the model has access to a background kg g , which is a subset of g with all the relations excluded from t mtr and t mte . ## our approach this section introduces our approach faan. given a meta-training set t mtr , the purpose of faan is to learn a metric function for predictions by comparing the input query to the given references. to achieve this goal, faan consists of three major parts: (1) adaptive neighbor encoder to learn adaptive entity representations; (2) transformer encoder to learn relational representations for entity pairs; tr m tr m tr m tr m tr m tr m 𝐳 " # 𝐳 @xmath0 *+, 𝐱 % *+, 𝐱 " *+, 𝐱 % ()( 𝐱 $ ()( ## experiments in this section, we conduct link prediction experiments to evaluate the performance of faan. ## conclusion this paper proposes an adaptive attentional network for few-shot kg completion, termed as faan. previous studies solve this problem by learning static representations of entities or references, ignoring their dynamic properties. faan proposes to encode entity pairs adaptively, and predict facts by adaptively matching references with queries. experiments on two public datasets demonstrate that our model outperforms current state-of-art methods with different few-shot sizes. our future work might consider other advanced methods to model few-shot relations, and exploiting more contextual information like textual description to enhance entity embeddings.