DTDES - KGE : Dual-Teacher Knowledge Distillation with Distinct Embedding Spaces for Knowledge Graph Embeddings (2025)

Abstract: Knowledge distillation for knowledge graph embedding (KGE) models effectively compresses KGE models by reducing their embedding dimensions. While existing methods distill knowledge from a high-dimensional teacher to a low-dimensional student, they typically rely on a single teacher embedding space, thereby overlooking valuable complementary knowledge from teachers in distinct embedding spaces. This paper introduces DTDES-KGE, a novel knowledge distillation framework that significantly enhances distillation performance by leveraging dual teachers in distinct embedding spaces. To overcome the challenge of spatial heterogeneity when integrating knowledge from dual teachers, we propose a spatial compatibility module for reconciliation. Additionally, we introduce a student-aware knowledge fusion mechanism to fuse the knowledge from dual teachers dynamically. Extensive experiments on two real-world datasets validate the effectiveness of DTDES-KGE.

## introduction knowledge graphs (kgs) represent factual knowledge with diverse structures as triples. their efficacy in organizing complex information has driven widespread adoption in various applications such as natural language processing @xcite , question answering @xcite , and recommendation systems @xcite . knowledge graph embedding (kge) models are pivotal for leveraging kgs, as they learn embeddings for entities and relations, facilitating the integration of kgs into numerous downstream applications @xcite @xcite . while diverse kge models have emerged, where euclidean space models prove adept at capturing chain structures and hyperbolic space models excel in representing hierarchies in kgs @xcite @xcite , achieving state-of-the-art performance often necessitates large embedding dimensions. this subsequently leads to prohibitive storage demands and increased inference latency, which severely limits their practical deployment, particularly in resource-constrained environments. knowledge distillation (kd) offers a promising avenue for compressing these large kge models, where a low-dimensional student model learns from a pre-trained, high-capacity teacher model by mimicking its output and embedding structure @xcite . however, existing knowledge distillation approaches for kges predominantly rely on teacher models trained within a single embedding space @xcite @xcite . this overlooks the rich, complementary knowledge that teachers from different embedding spaces, each of which excels at capturing distinct structural properties of kgs, could collectively provide. recognizing this untapped potential, and as illustrated in figure1, we propose to utilize dual teachers, one from euclidean space and another from hyperbolic space, to furnish a more comprehensive knowledge base for the student model. despite the considerable promise of fusing knowledge from euclidean and hyperbolic kge models @xcite , a critical challenge emerges: inter-space heterogeneity @xcite . the distinct geometric properties and scoring mechanisms inherent to these disparate spaces result in heterogeneous outputs and embedding structures. this fundamental incompatibility severely impedes effective knowledge fusion, preventing the student model from optimally learning from the combined wisdom of cross-space teachers. this paper investigates the pivotal research question: how can we effectively leverage dual teachers from distinct embedding spaces for kge model distillation? to this end, we introduce dtdes-kge, a novel dual teachers with distinct embedding spaces for knowledge graph embeddings distillation framework. dtdes-kge is designed to first reconcile spatial disparities and then intelligently fuse knowledge from dual teachers. specifically, to address the challenge of spatial heterogeneity, dtdes-kge incorporates a spatial compatibility module. this module employs optimal distribution selection to harmonize the heterogeneous outputs from the dual teachers and utilizes a predistillation phase to align their disparate embedding structures, thereby fostering better cross-space knowledge transfer. furthermore, to intelligently integrate the reconciled knowledge, dtdes-kge employs a student-aware knowledge fusion mechanism. this mechanism dynamically assigns adaptive weights to the dual teachers throughout the distillation process. it considers prior knowledge about teacher specializations, the knowledge divergence between teachers, and the student's current grasp of different knowledge facets, ensuring an optimal blend of guidance. we conduct comprehensive experiments on two real-world benchmark datasets. the results demonstrate that dtdes-kge significantly enhances the performance of low-dimensional kge models, enabling them to achieve results comparable or even superior to leading, larger kge models and existing distillation techniques. our main contributions are summarized as follows: (i) we propose dtdes-kge, a knowledge distillation framework for kge models utilizing dual teachers with distinct embedding spaces. (ii) we design a dedicated spatial compatibility module to mitigate inter-space heterogeneity and a student-aware knowledge fusion mechanism to dynamically and judiciously integrate complementary knowledge from these diverse teachers. (iii) experiment results show that dtdes-kge achieves comparable or superior performance to leading ap-proaches on two datasets. ## method this section first outlines kge model distillation preliminaries ( §3.1). as shown in figure 2 , our framework has three key components: a spatial compatibility module to reconcile heterogeneous dual-teacher knowledge ( §3.2); a student-aware mechanism to dynamically fuse this reconciled knowledge ( §3.3); and a combined strategy integrating logits-based ( §3.4) and feature-level distillation ( §3.5) into the overall objective ( §3.6). ## conclusion this paper introduced dtdes-kge, a novel knowledge distillation framework leveraging dual teachers from distinct embedding spaces to enhance kge distillation performance. it addresses teacher heterogeneity in output scores and embedding structures via a spatial compatibility module and dynamically fuses knowledge using a studentaware mechanism. extensive experiments on two real-world datasets validated dtdes-kge's effectiveness in improving student performance while compressing kge models.