Self-Distillation with Meta Learning for Knowledge Graph Completion (2022)

Abstract: In this paper, we propose a self-distillation framework with meta learning (MetaSD) for knowledge graph completion with dynamic pruning, which aims to learn compressed graph embeddings and tackle the long-tail samples. Specifically, we first propose a dynamic pruning technique to obtain a small pruned model from a large source model, where the pruning mask of the pruned model could be updated adaptively per epoch after the model weights are updated. The pruned model is supposed to be more sensitive to difficult-to-memorize samples (e.g., long-tail samples) than the source model. Then, we propose a one-step meta self-distillation method for distilling comprehensive knowledge from the source model to the pruned model, where the two models co-evolve in a dynamic manner during training. In particular, we exploit the performance of the pruned model, which is trained alongside the source model in one iteration, to improve the source model’s knowledge transfer ability for the next iteration via meta learning. Extensive experiments show that MetaSD achieves competitive performance compared to strong baselines, while being 10x smaller than baselines.

## introduction knowledge graphs (kg) have become the key technology to represent structural relations between entities and play an important role in question answering @xcite , dialogue systems @xcite , and entity disambiguation @xcite . however, most kgs are growing at a rapid pace and are far from complete. therefore, it is necessary to develop knowledge graph completion (kgc) approaches to add missing triples to the kgs, so as to improve the quality of kgs. recent advances in kgc primarily work on knowledge graph embedding (kge) by convert-* min yang and chengming li are corresponding authors. the dataset and code are publicly available at https: //github.com/pldlgb/metasd ing the entities and relations in kgs into lowdimensional vectors. early studies on kge introduce a margin-based pairwise ranking function to measure the euclidean distance or similarity between the relational projection of entities @xcite @xcite . among them, transe @xcite is the most widely used kge method, which views the relation as translation from a head entity to a tail entity. recently, neural networks, such as neural tensor network (ntn) @xcite and neural association model (nam) @xcite have been proposed to encode semantic matching and achieved remarkable predictive performance for kgc. to increase the capacity of the kge models, a larger embedding size with more parameters is a common technique in practice. as shown in figure 1 , the prediction performance of the kgc models such as dura @xcite and rp @xcite can be largely improved by increasing the graph embedding size. although the large graph embedding often bring obvious performance improvements, it may also become the major obstacle for model deployment and realtime prediction, especially for memory-limited and resource-constrained devices. in addition, the distribution of samples with longtail is prevalent in kgs @xcite , where a large portion of relations have much fewer triples than other relations. however, most previous stud-ies mainly focus on the predictive performance on overall test data, without taking long-tail samples into consideration. these models suffer from robust and generalization performance in the practical scenario. although several recent works @xcite have been proposed for few-shot kgc, these models are not adapted to the model compression frameworks. in this paper, we propose a self-distillation framework with meta learning (metasd) for knowledge graph completion with dynamic pruning. first, we propose a dynamic pruning technique to obtain a small pruned model from a large source model at the start of each training epoch. concretely, the pruning mask of the pruned model could be updated adaptively per epoch after updating the model weights. the pruned model is supposed to be more sensitive to the difficult-tomemorize samples (e.g., long-tail samples) than the source model. second, we propose a one-step meta self-distillation method to distill comprehensive knowledge from the source model to the pruned model, where the two models co-evolve in a dynamic manner during the whole knowledge distillation process. the key idea is to use the performance of the pruned model, which is trained alongside the source model in one iteration, to improve the source model for the next iteration by borrowing the idea of learning to learn from meta learning @xcite . in particular, we define the objectives of the source model as functions of the pruned model's performance on a quiz set. the usage of "gradient by gradient" strategy makes the source model adjust to the learning state of the pruned model, and improves both the source and pruned models. the main contributions of our method can be three-fold. (1) we propose a self-distillation framework to compress kg embeddings for kgc. the source and pruned models co-evolve in a dynamic manner during training, thus we can avoid pretraining a large model in advance, and the performance of the pruned model is not limited to that of a pre-trained large model. (2) we exploit the feedback from the pruned model to guide the source model with meta learning, making the source model transfer better knowledge to the pruned network. (3) experimental results on two benchmark datasets show that our model achieves competitive performance compared to strong baselines, while being 10x smaller than other kgc models. ## methodology problem definition suppose a kg can be viewed as a graph @xmath0, r, t)} ∈ e × r × e, where e and r represent the entity (node) set and relation (edge) set respectively. (h, r, t) represents a triple, where h, t and r indicate head entity, tail entity and the relation between two entities, respectively. given the kg g, the goal of kgc is to infer missing links based on existing triples in the kg. ## conclusion in this paper, we proposed a self-distillation framework with meta learning for graph embedding compression. concretely, we proposed a one-step meta self-distillation method for distilling comprehensive knowledge from the source model to the pruned model, where the two models co-evolved in a dynamic manner during training. experimental results showed that our model achieved competitive performance compared to strong baseline methods, while being 10x smaller than baseline methods.