Knowledge Base Embedding By Cooperative Knowledge Distillation (2020)

Abstract: Knowledge bases are increasingly exploited as gold standard data sources which benefit various knowledge-driven NLP tasks. In this paper, we explore a new research direction to perform knowledge base (KB) representation learning grounded with the recent theoretical framework of knowledge distillation over neural networks. Given a set of KBs, our proposed approach KD-MKB, learns KB embeddings by mutually and jointly distilling knowledge within a dynamic teacher-student setting. Experimental results on two standard datasets show that knowledge distillation between KBs through entity and relation inference is actually observed. We also show that cooperative learning significantly outperforms the two proposed baselines, namely traditional and sequential distillation.

## introduction knowledge bases (kb), organizing structured information about entities (nodes), and relations (edges) as graphs, are increasingly exploited as gold standard data sources for a broad range of human-ai tasks including language modeling @xcite , question answering @xcite and semantic search @xcite . although typical kbs may include a huge amount of observed knowledge through millions of entities and their relations, they are by nature incomplete since they can only capture a fraction of world knowledge. this limitation has given rise to extensive research work that focuses on the issue of predicting new knowledge from the observed one @xcite . this issue has been successfully tackled by neural approaches for representation learning of kbs @xcite @xcite . these models aim at representing kb entities and relations in low-dimensional embedding spaces, and supporting relational inferences using simple vector algebra. recent years have witnessed increasing interest toward embedding models leveraged to connect multiple kbs @xcite @xcite @xcite . the key objective of multi-graph representation learning is to empower the entity and relation models with different graph contexts that potentially bridge different semantic contexts. to achieve this goal, embeddings are learned upon the combined triples across graphs. although the above multi-graph representation learning methods have achieved promising results, they are still challenged by two main limitations. first, they are particularly suited to graph-alignment and machine translation as downstream tasks which necessarily leads to tackle computational challenges in large-scale kbs. second, these methods assume that each kb has access to all the entities and relations that are stored in the other kbs, while it may not be feasible neither relevant to the kbs to share unaligned information such as in personal kbs @xcite . following a different objective, we argue that apart from any downstream task, modeling the relational patterns across kbs might mainly focus on explicitly modeling connectivity patterns within each kb using its own observed triples and, infer additional patterns from its peer by only using partially aligned observed triples. as a motivating example let us consider two kbs (kb 1 and kb 2 ) that contain facts regarding cities, capitals, and countries as illustrated in figure 1 , but where none of them include the fact that rome is a city of italy. by learning an embedded space, kb 1 model may be able to correctly generalize the relation cityin and infer that rome is a city of italy by grouping other similar entity embeddings to rome such as the embedding of pisa. although this knowledge was not inferred by kb 1 , kb 2 can teach this information to kb 1 by distillation. then, kb 1 model will be able to understand the relation capitalof by directly observing examples within its own semantic context and the relation cityin by distilled knowledge from kb 2 semantic context. accordingly, unlike existing work in multi-graph embedding which relies on a unified view over multiple graphs, our work rather relies on multiple within-kb views that are bridged with aligned information. while each kb may be learning embeddings on its own semantic context based on associated hard triples, it can additionally exchange inferred knowledge from soft aligned triples provided by other kbs, in turn improving the embeddings of each other based on different semantic contexts. our key idea is to model a knowledge distillation process @xcite across kbs to empower their generalization ability. despite the number of works that show the rationale behind the entity and relation inference between kbs @xcite , none has shown the feasibility of the knowledge distillation framework to model knowledge inference between kbs. since the kbs play symmetric roles in knowledge transfer, one critical issue is how to train each kb model using entity/relation labels based on soft predictive distributions provided by the teacher, as well as its own predictive distribution. to tackle this issue, we argue for a mutual learning paradigm @xcite , where each kb acts dynamically as either a teacher or a student. unlike traditional static one-way knowledge transfer from a teacher model to a student model, we argue towards a two-way cooperative knowledge transfer between a kb and its peers. concretely our set up is the following: the representation learning model of each kb is equipped with two losses which are jointly optimized: 1) a classic kb supervised margin based ranking loss whose objective is to make the scores of positive triples lower than those of negative ones; and 2) a mimicry cooperative distillation loss that makes the posterior class predictions of aligned entities and aligned relations close to the entity and relation class probabilities of its peer respectively. through joint optimization, the knowledge is also naturally transferred from the seed aligned information to unaligned information. in summary, the main contributions of the paper are the following: 1) a first attempt to ground multigraph representation learning by a knowledge distillation theoretical framework; 2) a novel kb representation learning model called kd-mkb, based on a cooperative knowledge distillation strategy; 3) experiments on two standard datasets, wn18rr and fb15k-237, that empirically validate the rationale of knowledge distillation across kbs and show the effectiveness of the cooperative knowledge distillation as proposed in kd-mkb. the remainder of this paper is structured as follows. section 2 presents the related works. in section 3, we first introduce the used preliminary notions and then detail the kd-mkb model. in section 4, we present and discuss the experimental results. section 5 concludes the paper. ## cooperative knowledge distillation across multiple kbs in this section, we describe kd-mkb, a kb representation learning model. we first introduce a couple of terminological definitions so we can formally define our model. a pair (kb i , kb j ) ∈ kb 2 , kb i (resp. kb j ) comprises a set of e i (resp. e j ) entities, a set of r i (resp. r j ) relations. i e (i, j) = {(e i ## experiments two main objectives have guided our experiments: 1) show the validity of knowledge distillation to formally support knowledge transfer between kbs; 2) evaluate the effectiveness of the kd-mkb model. ## conclusion and future work this paper presents a new framework for learning entity and relation embeddings over multiple kbs. our framework exploits a new way to transfer learning from one kb model to its peers. first, we formalize entity and relation inference between kbs as a distillation loss over posterior probability distributions on aligned knowledge. grounded on this finding, we propose and formalize a cooperative distillation framework where a set of kb models are jointly learned by using each of them hard labels from their own context and also soft labels provided by peers. we empirically demonstrate the rationale behind knowledge distillation between kbs and show the effectiveness of our cooperative learning framework on the link prediction task compared to the existing distillation strategies. further experiments are planned for future work using more complex and realistic configurations for multiple kbs learning to assess about the generalizability of our findings. we also plan to extend our approach to consider weak alignments while distilling knowledge over pairs of kbs. this would empower the cooperative learning with higher generalization ability particularly for heterogeneous kbs. we believe that this work could be used without major change in the core methodology to support a wide range of knowledge-driven applications.