Context-Aware Adapter Tuning for Few-Shot Relation Learning in Knowledge Graphs (2024)

Abstract: Knowledge graphs (KGs) are instrumental in various real-world applications, yet they often suffer from incompleteness due to missing relations. To predict instances for novel relations with limited training examples, few-shot relation learning approaches have emerged, utilizing techniques such as meta-learning. However, the assumption is that novel relations in meta-testing and base relations in meta-training are independently and identically distributed, which may not hold in practice. To address the limitation, we propose RelAdapter, a context-aware adapter for few-shot relation learning in KGs designed to enhance the adaptation process in meta-learning. First, RelAdapter is equipped with a lightweight adapter module that facilitates relation-specific, tunable adaptation of meta-knowledge in a parameter-efficient manner. Second, RelAdapter is enriched with contextual information about the target relation, enabling enhanced adaptation to each distinct relation. Extensive experiments on three benchmark KGs validate the superiority of RelAdapter over state-of-the-art methods.

## introduction knowledge graphs (kgs) @xcite @xcite have been widely adopted to describe realworld facts using triplets in the form of (head entity, relation, tail entity). however, curating and maintaining all the possible ground-truth triplets is impossible, and various approaches for knowledge graph completion @xcite @xcite have been proposed to discover missing facts. many of these methods adopt a supervised learning paradigm, which require abundant training data for each relation. in real-world settings, novel and emerging relations, along with many relations in the long tail, are associated with very few instances @xcite , limiting their performance. subsequently, few-shot relation learning (fsrl) on kgs has emerged to handle novel relations with only a few known instances. an established line of work @xcite employs meta-learning, most notably model-agnostic meta-learning (maml) @xcite . maml aims to learn a prior from a series of meta-training tasks, which can be rapidly adapted to downstream meta-testing tasks. the meta-training tasks are specifically constructed in a few-shot setup to mimic downstream tasks. in the context of fsrl, each task contains only a few training instances of a single relation, and the objective is to predict more instances for a novel task (i.e., relation foot_0 ) not seen in meta-training. for example, metar @xcite aims to learn a relation-specific prior (also called meta-knowledge) during the meta-training stage, using a series of meta-training tasks constructed from a set of base relations with abundant instances. subsequently, the meta-knowledge is leveraged for rapid adaptation, through a lightweight fine-tuning step, for few-shot predictions on novel relations in metatesting. limitation of prior work. the major limitation of fsrl methods based on meta-learning lies in the assumption that the meta-training and metatesting tasks are independently and identically distributed (i.i.d.). however, different relations may diverge significantly in their underlying distributions, thereby weakening the i.i.d. task assumption. to investigate this hypothesis, we randomly sample a large number of relation pairs from standard benchmark datasets, namely, wiki, fb15k-237 and umls foot_1 . we perform a mean pooling across all entities within each relation task to derive an average embedding as the task representation. for every pair of relations, we plot their cosine similarity in fig. 1 . the results reveal a wide variance in the similarities between relations, suggesting that a uniform adaptation process may not suffice for all relations. in particular, for out-of-distribution relations, performance degradation is generally anticipated @xcite . challenges and insights. specifically, we identify two open challenges, at the model and data levels, towards more universal downstream adaptation. at the model level, the first challenge (c1) is how to design a relation-specific adaptation module that can be tailored to each relation, while still leveraging the meta-knowledge learned from metatraining. the goal is to enable a more flexible adaptation process that allows for a relation-specific balance between the global prior and local task input. at the data level, the second challenge (c2) is how to augment few-shot relation instances during meta-testing in an unsupervised manner, thereby further enhancing adaptation to novel relations. to address these challenges, we introduce a context-aware adapter framework, called re-ladapter, for few-shot relation learning. to overcome the model-level challenge (c1), we draw inspiration from parameter-efficient fine-tuning @xcite @xcite . specifically, we propose to integrate a lightweight adapter module into the meta-learning framework. the adapter module enables a relation-specific, tunable adaptation of the global prior to suit the local task in the metatesting stage. for the data-level challenge (c2), we propose to inject additional contextual information about the target relation into meta-testing. the contextual information is extracted based on existing kg structures without requiring any extra annotation, serving as a form of data augmentation to enrich the few-shot relation instances. this strategy endows the adapter with more relation-specific contexts, and the context-aware adapter can enhance relation-specific adaptation. contributions. in summary, we make the follow- ## preliminaries in this section, we introduce the problem of few-shot relation learning (fsrl) and the metalearning framework for this problem. problem formulation. a knowledge graph @xmath0, r) comprises a set of triplets. each triplet is represented by the form (h, r, t) for some h, @xmath1, where v is the set of entities and r is the set of relations. the goal is to predict the missing tail entities in a query set, q @xmath2, r, ?) | @xmath3, 2, . . . , }, w.r.t. the given h @xmath4. in the paper, for each triplet in (h j , r, ?) ∈ q r , a type-constrained candidate set c h j ,r is provided and the objective is to rank the true tail entities highest among the candidates. together, the support and query sets form a task t @xmath5, q r ) for r. the support set provides a few training instances, known as k-shot relation learning, where |s r | = k is typically a small number. meta-learning framework. given the success of meta-learning in few-shot problems, we adopt metar @xcite , a popular meta-learningbased approach for fsrl, as our learning framework. metar consists of two stages: meta-training and meta-testing, aiming to learn a prior φ from the meta-training stage that can be adapted to the meta-testing stage. on one hand, meta-training involves a set of seen relations r tr , and operates on their task data d @xmath6. on the other hand, meta-testing involves a set of novel relations r te such that r tr ∩ r @xmath7, and operates on their task data d @xmath8. note that the ground-truth tail entities are provided in the query sets of the meta-training tasks d tr , whereas the objective is to make predictions for the query sets of the meta-testing tasks d te . meta-training. during meta-training, the model learns a prior φ, which serves as a good initialization to extract a shared relation meta, r @xmath9, for each task t @xmath10, q r ) ∈ d tr . specifically, the relation meta r tr is obtained through a mean pooling of all (head, tail) pairs in the support set of the task t r , as follows. it is worth noting that the prior φ initializes the relation-meta learner rml, which further generates the relation meta r tr . subsequently, r tr is rapidly updated by a gradient meta g tr calculated from the loss on the support set, i.e., r ′ @xmath11, where β is the step size. the resulting r ′ tr serves as the relation meta adapted to the support set, which is used to calculate the loss on the query set. finally, the query loss is backpropagated to update the prior φ and the embedding matrix emb for the entities. more specifically, the support and query losses are calculated as follows. here, s(•, •) is a scoring function such as transe @xcite , i.e., s(h, r, t) = ∥h+r-t i ∥, where ∥ • ∥ denotes the l 2 norm. emb is an entity embedding matrix, which is initialized by a pretrained model and can be further optimized during meta-training. (h, r, t ′ ) refers to a negative triplet of the relation r, where t ′ is randomly sampled from the type-constrained candidate set. γ represents the margin which is a hyper-parameter. ## methodology in this section, we introduce the proposed approach reladapter. as depicted in fig. 2 , reladapter has two important components, namely, (a) adapter network and (b) entity context. on one hand, the adapter network aims to facilitate relation-specific and tunable adaptation of meta-learned prior in a parameter-efficient manner. on the other hand, the entity context aims to enrich the adapter with contextual information pertinent to the target relation, enabling more precise adaptation to each distinct relation. the two components are integrated into a context-aware adapter, which enhances fsrl for the novel relations in meta-testing. in the rest of the section, we first introduce the context-aware adapter, followed by the details of the meta-training and meta-testing stages. ## conclusion in this paper, we proposed reladapter, a contextaware adapter for few-shot relation learning (fsrl). we investigated the limitation of fsrl methods in prevailing meta-learning frameworks, which rely on the i.i.d. assumption. this assumption may not hold for novel relations with distribution shifts from the seen relations. based on this insight, we introduced a context-aware adapter module, enabling relation-specific, tunable and fine-grained adaptation for each distinct relation. extensive experiments were conducted on three benchmark datasets, demonstrating the superior performance of reladapter.