D yna S emble: Dynamic Ensembling of Textual and Structure-Based Models for Knowledge Graph Completion (2024)

Abstract: We consider two popular approaches to KnowledgeGraph Completion (KGC): textual modelsthat rely on textual entity descriptions, andstructure-based models that exploit the connectivitystructure of the Knowledge Graph(KG). Preliminary experiments show that theseapproaches have complementary strengths:structure-based models perform exceptionallywell when the gold answer is easily reachablefrom the query head in the KG, while textualmodels exploit descriptions to give goodperformance even when the gold answer isnot easily reachable. In response, we proposeDynaSemble, a novel method for learningquery-dependent ensemble weights to combinethese approaches by using the distributions ofscores assigned by the models in the ensembleto all candidate entities. DynaSemble achievesstate-of-the-art results on three standard KGCdatasets, with up to 6.8 pt MRR and 8.3 ptHits@1 gains over the best baseline model forthe WN18RR dataset.

## introduction the task of knowledge graph completion (kgc) can be described as inferring missing links in a knowledge graph (kg) based on given triples (h, r, t), where r is a relation that exists between the head entity h and the tail entity t. several kgc approaches, such as nbfnet @xcite and @xcite , exploit the underlying graph structure, often using gnns. on the other hand, textual models such as simkgc @xcite and hitter @xcite leverage pre-trained large language models (llms) such as bert @xcite to utilize textual descriptions of the kg entities and relations for kgc. our preliminary experiments suggest that when the gold answer t for query (h, r, ?) is reachable from h via a path of reasonable length in the kg, structure-based models tend to outperform textual models. in contrast, textual models use textual de-scriptions to perform better than structure-based models when t is not easily reachable from h. motivated by our findings, we seek to explore how ensembling, an approach currently underrepresented in kgc literature (see @xcite for an example), can effectively harness the complementary strengths of these models. consequently, we propose dynasemble: a novel, simple, model-agnostic and lightweight method for learning ensemble weights such that the weights are (i) query-dependent and (ii) learned from statistical features obtained from the distribution of scores assigned by individual models to all candidate entities. this approach results in a new state-of-the-art baseline when applied on two strong kgc models: simkgc and nbfnet, which are textual and structure-based in nature, respectively. on three kgc datasets, we find that applying dynasemble to simkgc and nbfnet consistently improves kgc performance, outperforming best individual models by up to 6.8 pt mrr and 8.3 pt hits@1 on the wn18rr dataset. to the best of our knowledge, our results are state of the art for all three datasets. further experiments (including a fourth dataset to which nbfnet does not scale) show that dynasemble generalises to ensembling with another kg embedding model, @xcite , with similar gains. we also demonstrate that dynasemble outperforms conventional model-combination techniques such as static ensembling (where the ensemble weight is a tuned constant hyperparameter) and re-ranking. we release all code foot_0 to guide future research. overview of related work: we focus on three types of kgc models. the first type consists of graph neural network (gnn) based models such as nbfnet @xcite , @xcite ) that leverage neighborhood information to train distinct gnn architectures. the second type contains textual models such as kg-bert @xcite , hitter @xcite and @xcite which fine-tune a pre-trained llm on textual descriptions of entities and relations for kgc. the third type involves models such as @xcite and @xcite ) that learn low-dimensional embeddings for entities and relations and compose them by employing unique scoring functions. unification of these approaches has not been extensively studied in kgc literature. vem2l @xcite proposes a method to encourage multiple kgc models to learn from each other during training. @xcite finds that their textual model struggles when the query has a large number of correct answers in the training set and routes those queries to a structurebased model as a consequence, exhibiting some performance gains. since our main experiments are based on nbfnet, simkgc, hitter and ro-tate, we describe these next. ## dynasemble our goal is to dynamically ensemble k kgc models m i , which may be textual or structure-based, to maximize performance. each model m i assigns a score m i (h, r, t) to all candidate tails @xmath0, r, ?). these models are trained independently and their parameters are frozen before ensembling. we formulate the ensemble e as: the scores obtained after normalization lie in the range [0,1] for all models. we next describe the simple model used to learn the query-dependent ensemble weights w i . model: we extract the following features from the score distribution of each model m i : in the above equations, mean() and var() are the standard mean and variance functions respectively, 57.0 51.9 67.3 42.4 32.8 60.9 -----whose outputs are concatenated to obtain the feature. this choice is driven by the insight that the variance and mean of the distribution of scores computed by any model over e is correlated to the model confidence. a more detailed discussion, along with an exploration of other possible feature sets can be found in appendix c. next, we concatenate these features for all m i to obtain a final feature vector that is passed to an independent 2-layer mlp (mlp i ) for each model m i to learn query-dependent w i : intuitively, this concatenation informs each mlp about the relative confidence of all models regarding their predictions, enhancing the ensemble weight computation for corresponding models. note that our approach is agnostic to models m i . our experiments in this paper mostly involve only one textual model. therefore, we learn the ensemble weights for the other models with respect to this textual model, which is assigned a fixed weight of 1. this decreases the parameter count while still being as expressive as learning distinct ensemble weights for all models. the method for learning these other weights is unchanged. loss function: we train dynasemble on the validation set (traditionally used to tune ensemble weights) using margin loss between the score of the gold entity and a set of negative samples. the train set is not used since all models are likely to give high-confidence predictions on its triples (appendix d). if the gold entity is t * and the set of negative samples is n, the loss function l for query @xmath1, r, ?) is: ## analysis we perform four further analyses to answer the following questions: q1. how does the behavior of textual and structure-based models vary with reachability? q2. do the weights learned by dynasemble follow expected trends with reachability? q3. does dynasemble improve performance over conventional model-combination techniques? q4. how does dynasemble of a textual and structure-based model compare to dynasemble of two textual or structure-based models? reachability ablation: to answer q1, we divide the test set for each dataset into 'reachable' and 'unreachable' splits. a triple (h, r, t) is part of the reachable split if t can be reached from h with a path of length at most l (= 2) in the kg. if not, it is put in the unreachable split. we present splitwise results for [nbf], [sim] and [sim]+[nbf] on the wn18rr and fb15k-237 datasets in ## conclusion and future work we present dynasemble: a simple, novel, modelagnostic and lightweight dynamic ensembling approach for kgc, while also highlighting the complementary strengths of textual and structure-based kgc models. our state-of-the-art results for a dynasemble of simkgc and nbfnet over three standard kgc datasets (wn18rr, fb15k-237 and codex-m) creates a new competitive ensemble baseline for the task. we release all code for future research. future work includes tighter trainingtime unification methods, and extensions to temporal @xcite and multilingual kgc models @xcite .