To Copy Rather Than Memorize: A Vertical Learning Paradigm for Knowledge Graph Completion (2023)

Abstract: Embedding models have shown great power in knowledge graph completion (KGC) task. By learning structural constraints for each training triple, these methods implicitly memorize intrinsic relation rules to infer missing links. However, this paper points out that the multi-hop relation rules are hard to be reliably memorized due to the inherent deficiencies of such implicit memorization strategy, making embedding models underperform in predicting links between distant entity pairs. To alleviate this problem, we present Vertical Learning Paradigm (VLP), which extends embedding models by allowing to explicitly copy target information from related factual triples for more accurate prediction. Rather than solely relying on the implicit memory, VLP directly provides additional cues to improve the generalization ability of embedding models, especially making the distant link prediction significantly easier. Moreover, we also propose a novel relative distance based negative sampling technique (ReD) for more effective optimization. Experiments demonstrate the validity and generality of our proposals on two standard benchmarks. Our code is available at https://github.com/rui9812/VLP .

## introduction knowledge graphs (kgs) structurally represent human knowledge as a collection of factual triples. each triple (h, r, t) represents that there is a relation r between head entity h and tail entity t. with the massive human knowledge, kgs facilitate a myriad of downstream applications @xcite . however, real-world kgs such as freebase @xcite are far from complete @xcite . this motivates substantial research on the knowledge graph completion (kgc) task, i.e., automatically inferring missing triples. as an effective solution for kgc, embedding model learns representations of entities and relations with pre-designed relation operations. for example, transe @xcite represents relations as translations between head and tail entities. rescal @xcite , distmult @xcite and complex @xcite model the three-way interactions in each triple. rotate @xcite , quate @xcite and duale @xcite represent relations as rotations in different dimensions. rot-pro @xcite further introduces the orthogonal projection for each relation. essentially, embedding models learn structural constraints for every factual triple during the training period. for example, for each training triple (h, r, t), transe constrains that the head embedding h plus the relation embedding r equals the tail embedding t. such single-triple constraints empower embedding models to implicitly perceive (i.e., memorize) the high-order entity connections and intrinsic relation rules @xcite . as shown in figure 1 , by imposing the structural constraints (e.g., h + @xmath0, embedding models can memorize the entity connection (x, r 1 ∧ r 2 , z) and the relation rule r 1 ∧ r 2 → r. in this way, the missing link model score function g(wr,1h + br, wr,2t) space wr,1 br wr,2 g(q, k) rescal @xcite ) h ⊤ wrt i 0 wr q ⊤ k r transe @xcite -∥h + r -t∥ i r i -∥q -k∥ r transr @xcite -∥wrh + r -wrt∥ wr r wr -∥q -k∥ r distmult @xcite h ⊤ diag(r)t diag(r) 0 i q ⊤ k r complex @xcite re(h ⊤ diag(r)t) diag(r) 0 i re(q ⊤ k) c rotate @xcite -∥h • r -t∥ diag(r) 0 i -∥q -k∥ c we give an affirmative answer by presenting vertical learning paradigm (vlp), which endows embedding models with the ability to explicitly consult related factual triples (i.e., vertical references) for more accurate prediction. specifically, to answer (h, r, ?), vlp first selects n relevant reference queries in the training graph, and then treats their ground-truth entities as the reference answers for embedding models to jointly predict the target t. this learning process can be viewed as an explicit copy strategy, which is different from the implicit memorization strategy of hlp, making it significantly easier to predict distant links. moreover, to effectively optimize the models, we further propose a novel relative distance based negative sampling technique (red), which can generate more informative negative samples and reduce the toxicity of false negative samples. note that vlp and red are both general techniques and can be widely applied to various embedding models. our contributions are summarized as follows: ## preliminaries to elicit our proposal from a general paradigm perspective, we give a bird's eye view of existing embedding models in this section. we first review the problem setup of kgc task. afterwards, we summarize a generalized score function of embedding models and describe how the models learn to predict new links (i.e., horizontal learning paradigm). ## motivation the motive of our work originates from an observation that embedding models underperform in predicting links between distant entity pairs (refer to appendix a for more details). since the effectiveness of embedding models is largely determined by the ability to learn intrinsic relation rules @xcite @xcite , such inferior performance reveals that the models are hard to memorize the multi-hop relation rules. we attribute this deficiency to the multi-hop bias accumulation and high-demanding memory capacity in the implicit memorization strategy of hlp. ## related work embedding models can be roughly categorized into distance based models and semantic matching models @xcite . distance based models use the euclidean distance to measure the plausibility of each triple. a series of work is conducted along this line, such as transe @xcite transh @xcite , transr @xcite , rotate @xcite , pairre @xcite , rot-pro @xcite , reflecte @xcite and so on. transe and rotate are the most representative distance-based models, which represent relations as translations and rotations, respectively. semantic matching models utilize multiplicative functions to score each triple, including rescal @xcite , distmult @xcite , com-plex @xcite , quate @xcite , duale @xcite and so on. typically, rescal @xcite defines each relation as the tensor decomposition matrix. dist-mult @xcite simplifies the relation matrices to be diagonal for preventing overfitting. however, existing embedding models essentially follow the horizontal learning paradigm, underperforming in predicting links between distant entities. moreover, some advanced techniques are proposed to improve embedding models, such as graph encoders @xcite @xcite and regularizers @xcite . note that our proposals are orthogonal to these techniques, and one can integrate them for better performance. ## conclusion in this paper, we present a novel learning paradigm named vlp for kgc task. vlp can be viewed as an explicit copy strategy, which allows embedding models to consult related triples for explicit references, making it much easier to predict distant links. moreover, we also propose red, a new negative sampling technique for more effective optimization. the in-depth experiments on two datasets demonstrate the validity and generality of our proposals.