D isentangle-based C ontinual G raph R epresentation L earning (2020)

Abstract: Graph embedding (GE) methods embed nodes (and/or edges) in graph into a low-dimensional semantic space, and have shown its effectiveness in modeling multi-relational data. However, existing GE models are not practical in real-world applications since it overlooked the streaming nature of incoming data. To address this issue, we study the problem of continual graph representation learning which aims to continually train a GE model on new data to learn incessantly emerging multi-relational data while avoiding catastrophically forgetting old learned knowledge. Moreover, we propose a disentangle-based continual graph representation learning (DiCGRL) framework inspired by the human’s ability to learn procedural knowledge. The experimental results show that DiCGRL could effectively alleviate the catastrophic forgetting problem and outperform state-of-the-art continual learning models. The code and datasets are released on https://github.com/KXY-PUBLIC/DiCGRL .

## introduction multi-relational data represents relationships between entities in the world, which is usually denoted as a multi-relational graph with nodes and edges connecting them. it is widely used in realworld nlp applications such as knowledge graphs (kgs) (e.g., freebase @xcite and dbpedia @xcite ) and information networks (e.g., social network and citation network). therefore, modeling multi-relational graph with graph embeddings @xcite @xcite has been attracting intensive attentions in both academia and industry. graph embedding (ge), aiming to embed nodes and/or edges in the graph into a low-dimensional semantic space to enable neural models to effectively and efficiently utilize multi-relational data, has demonstrated remarkable effectiveness in various downstream nlp tasks such as question answering @xcite and dialogue system @xcite . nevertheless, most existing graph embedding works overlook the streaming nature of the incoming data in real-world scenarios. in consequence, these models have to be retrained from scratch to reflect the data change, which is computationally expensive. to tackle this issue, we propose to study the problem of continual graph representation learning (cgrl) in this work. the goal of continual learning is to alleviate catastrophically forgetting old data while learning new data. there are two mainstream continual learning methods in nlp: (1) consolidationbased methods @xcite @xcite which consolidate the important model parameters of old data when learning new data; and (2) memory-based methods @xcite @xcite which remember a few old examples and learn them with new data jointly. despite the promising results these methods have achieved on classification tasks, their effectiveness has not been validated on graph representation learning. unlike the classification problem where instances are generally independent and can be operated individually, nodes and edges in multi-relational data are correlated, making it sub-optimal to directly deploy existing continuous learning methods on the multi-relational data. in cognitive psychology @xcite , procedural knowledge refers to a set of operational steps. its smallest unit is production, where multiple productions can complete a series of cognitive activities. when learning new procedural knowledge, humans would update cognitive results by only updating a few related productions and leave the rest intact. intuitively, such a process can be mimicked to learn constantly growing multirelational data by regarding each new data as a new procedural knowledge. for example, as illustrated in figure 1 , the relational triplets of barack obama and michelle obama are related to three concepts: "family", "occupation" and "location". when a new relational triplet (michelle obama, daughter, malia ann obama) appears, we only need to update the "family"-related information in barack obama. consequently, we can further infer that the triplet (barack obama, daughter, malia ann obama) also holds. we conduct extensive experiments on both kge and ne settings based on the real-world scenarios, and the experimental results show that dicgrl effectively alleviates the catastrophic forgetting problem and significantly outperforms existing continual learning models while remaining efficient. ## experiments in this section, we evaluate our model on two popular tasks: link prediction for knowledge graph and node classification for information network. ## conclusion and future work in this paper, we propose to study the problem of continual graph representation learning, aiming to handle the streaming nature of the emerging multi-relational data. to this end, we propose a disentangled-based continual graph representation learning (dicgrl) framework, inspired by human's ability to learn procedural knowledge. extensive experiments on several typical kge and ne datasets show that dicgrl achieves consistent and significant improvement compared to existing continual learning models, which verifies the effectiveness of our model on alleviating the catastrophic forgetting problem. in the future, we will explore to extend the idea of disentanglement in the continual learning of other nlp tasks.