Relation-aware Ensemble Learning for Knowledge Graph Embedding (2023)

Abstract: Knowledge graph (KG) embedding is a fundamental task in natural language processing, and various methods have been proposed to explore semantic patterns in distinctive ways. In this paper, we propose to learn an ensemble by leveraging existing methods in a relation-aware manner. However, exploring these semantics using relation-aware ensemble leads to a much larger search space than general ensemble methods. To address this issue, we propose a divide-search-combine algorithm RelEns-DSC that searches the relation-wise ensemble weights independently. This algorithm has the same computation cost as general ensemble methods but with much better performance. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed method in efficiently searching relation-aware ensemble weights and achieving state-of-the-art embedding performance. The code is public at https://github.com/LARS-research/RelEns.

## introduction knowledge graph (kg) embedding is a popular method for inferring latent features and making predictions in incomplete kgs @xcite . this technique involves transforming entities and relations into low-dimensional vectors and using a scoring function @xcite to assess the plausibility of a triplet (consisting of a head entity, a relation, and a tail entity). well-known scoring functions, such as transe @xcite , complex @xcite , conve @xcite , and compgcn @xcite , have demonstrated remarkable success in learning from kgs. ensemble learning is a well-known technique that improves the performance of machine learning tasks by combining and reweighting the predictions of multiple models @xcite ; wolpert, 1 l. yue and y. zhang made equal contributions, correspondence is to q. @xcite . its effectiveness has also been verified in kg embedding by previous studies @xcite @xcite . while designing different scoring functions to model various relation properties @xcite @xcite , such as symmetry, inversion, composition and hierarchy, is crucial for achieving good performance, existing ensemble methods do not reflect the relation-wise characteristics of different models. this motivates us to propose specific ensemble weights for different relations, named as relens problem, in this paper. by doing so, different kg embedding models can specialize in different relations, leading to improved performance. however, the number of parameters to be searched will linearly increase, which can significantly complicate the ensemble construction process especially for kgs with many relations. to alleviate the difficulty of searching for relation-wise ensemble weights, we propose dsc, an algorithm that divide the overall ensemble objective into multiple sub-problems, search for the ensemble weights for each relation independently, and then combine the results. this approach significantly reduces the size of the search space and evaluation cost for individual sub-problems, compared to the overall objective. in summary, we propose relens-dsc, a novel relation-aware ensemble learning method for kg embedding that searches different ensemble weights independently for different relations, using a divide-concur strategy. empirically, relens-dsc significantly improves the performance on three benchmark datasets (wn18rr, fb15k-237, nell-995) and achieves the first place on the largescale leaderboards ogbl-biokg and ogbl-wikikg2. our approach is more effective than general ensemble techniques, and it is more efficient with the divide-concur strategy under parallel computing. ## proposed method denote a kg as @xmath0, r, d), where v contains v entities (nodes), r contains r types of relations between entities, and @xmath1, r, t) : h, @xmath2, @xmath3. d is split into three disjoint sets d tra , d val , d tst for training, validation and testing, respectively. the learning objective of a kg embedding model is to rank positive triplets higher than negative triplets, in order to accurately identify the potential positive triplets missed in the current graph @xcite . specifically, formulated as a tail prediction problem 2 , the kg embedding model aims to rank the tail entity t of a given triplet @xmath4, r, t), which belongs to either d val or d tst , higher than a set of negative entities. the set of negative entities is defined as n @xmath5: (h, r, e) / ∈ d}. the model f (x) computes a score vector s for each entity @xmath6, which indicates the degree of plausibility that the triplet (h, r, e) is true. a ranking function γ s is used to convert the scores s into a ranking list @xmath7, . . . , p c ) for the @xmath8. a smaller rank value implies the higher prediction priority. following @xcite @xcite , we adopt mean reciprocal ranking (mrr) as the evaluation metric. larger mrr indicates better performance. ## experiments the experiments were implemented using python and run on a 24gb nvidia gtx3090 gpu. as the ranking function γ(•) and mrr are nondifferentiable, we chose the widely used bayesian optimization technique, tree-parzen estimator (tpe) @xcite , to solve the maximization problems in eq. ( 1 ) and eq. ( 2 ), the details of which are provided in the appendix b.3. ## conclusion this paper introduces a novel ensemble method, relation-aware ensemble with divide-search-combine (relens-dsc) for kg embedding. the proposed relens-dsc learns relation-specific ensemble weights for different models and efficiently searches the weights using the divide-concur strategy. empirical results demonstrate that our proposed method outperforms existing ensemble methods for kg embedding, in both effectiveness and efficiency. limitations. the proposed method mainly addresses the ensemble problem for entity prediction tasks in knowledge graph completion. however, it does not effectively address the other graph learning tasks, such as entity/node classification, relation prediction, and graph classification. in addition, the significance of relens-dsc is under the case of multi-relational graphs like knowledge graph and heterogeneous graph, thus is not well adapted to homogeneous graph with single edge type.