Knowledge Graph Embedding with Atrous Convolution and Residual Learning (2020)

Abstract: Knowledge graph embedding is an important task and it will benefit lots of downstream applications. Currently, deep neural networks based methods achieve state-of-the-art performance. However, most of these existing methods are very complex and need much time for training and inference. To address this issue, we propose a simple but effective atrous convolution based knowledge graph embedding method. Compared with existing state-of-the-art methods, our method has following main characteristics. First, it effectively increases feature interactions by using atrous convolutions. Second, to address the original information forgotten issue and vanishing/exploding gradient issue, it uses the residual learning method. Third, it has simpler structure but much higher parameter efficiency. We evaluate our method on six benchmark datasets with different evaluation metrics. Extensive experiments show that our model is very effective. On these diverse datasets, it achieves better results than the compared state-of-the-art methods on most of evaluation metrics. The source codes of our model could be found at https://github.com/neukg/AcrE .

## introduction knowledge graph is a kind of valuable knowledge bases and it is important for many ai-related applications. generally, a kg stores factual knowledge in the form of structural triplets @xmath0, r, t>, which means there is a kind of r relation from h (head entity) to t (tail entity). nowadays, great achievements have been made in building large scale kgs. usually a kg may contain millions of entities and billions of relational facts. however, there are still two major difficulties that prohibit the availability of kgs. first, although most existing kgs contain large amount of triplets, they are far from completeness. second, most existing kgs are stored in symbolic and logical formations while applications often involve numerical computing in continuous spaces. to address these two issues, researchers proposed knowledge graph embedding (kge) methods that aim to learn a kind of embedding representations for a kg's items (entities and relations) by projecting these items into some continuous low-dimensional spaces. generally, different kinds of kge methods mainly differ in how to view the role of relations in the projected spaces. for example, translation based methods (transe @xcite , transh @xcite , transr @xcite , transd @xcite , et al.) view the relation in a triplet as a translation operation from the head entity to the tail entity. other kge methods view relations as some kind of combination operators that link head entities and tail entities. for example, hole @xcite employs a circular correlation function as the combination operator in the project space. complex @xcite makes use of complex valued embeddings and takes the matrix decomposition as the combination operator. rt @xcite uses tucker decomposition for kge. rotatee @xcite use the rotation operation in the complex space as the combination operator. experimental results show these existing methods have strong feasibility and robustness in solving the mentioned two issues. recently, deep neural networks (dnn) based kge methods @xcite @xcite @xcite push the performance of kge to a soaring height. compared with previous methods, this kind of methods can learn more effective embeddings mainly due to the powerful learning ability inherent in the dnn models. however, as pointed out by @xcite that existing research did not make a proper trade-off between the model complexity (the number of parameters) and the model expressiveness (the performance in capturing semantic information). thus deep convolutional neural networks (dcnn) based methods are achieving more and more research attention due to their simple but effective structure. however, @xcite point out that the dcnn based methods usually suffer from the reduced feature resolution issue that is caused by the repeated combination of max-pooling and down-sampling("striding") performed at consecutive layers of dcnns. this will result in feature maps with significantly reduced spatial resolution when dcnn is employed in a fully convolutional fashion. to address this issue, we propose an atrous convolution based kge method which allows the model to effectively enlarge the field of view of filters almost without increasing the number of parameters or the amount of computations. to address the vanishing/exploding gradient issue inherent in the dnn based learning frame and the original information forgotten issue when more convolutions used, we introduce residual learning in the our method. we propose two learning structures to integrate different kinds of convolutions together: one is a serial structure, and the other is a parallel structure. we evaluate our method on six diverse benchmark datasets. extensive experiments show that our method achieves better result than the compared state-of-the-art baselines under most evaluation metrics on these datasets. ## related work translation based kge methods view the relation in a triplet as a translation operation from the head entity to the tail entity. these methods usually define a score function (or energy function) that has a form like ||h + rt || to measure the plausibility of a triplet. during training, almost all of them minimize a margin based ranking loss function over the training data. transe @xcite ) is a seminal work in this branch. it directly takes the embedding space as a translation space. formally, it tries to let h + r ≈ t @xmath1, r, @xmath2. transh @xcite models a relation as a hyperplane together with a translation operation on it. transr @xcite ) models entities and relations in distinct spaces, i.e., the entity space and multiple relation spaces. @xcite models each entity or relation by two vectors. transparse @xcite mainly considers the heterogeneity property and the imbalance property in kgs. ptranse @xcite integrates relation paths into a transe model. itransf @xcite uses a sparse attention mechanism to discover hidden concepts of relations and to transfer knowledge through the sharing of concepts. recently, researchers also employ the methods of combining different distance functions together for kge. for example, @xcite proposed a multi distance embedding (mde) model, which consists of several distances as objectives. bilinear kge models use different kinds of combination operators other than the translation. for example, hole @xcite employs a circular correlation as the combination operator. complex @xcite makes use of complex valued embedding and takes matrix decomposition as the combination operator. similar to complex, rotate @xcite also use a complex space where each relation is defined as a rotation from the source entity to the target entity. @xcite proposed dihedral for kg relation embedding. by leveraging the desired properties of dihedral group, their method could support many kinds of relations like symmetry, inversion, etc. @xcite propose the relational tucker3(rt) decomposition for multi-relational link prediction in knowledge graphs. other work, kg2e @xcite uses a density-based method to model the certainty of entities and relations in a space of multi-dimensional gaussian distributions. transg @xcite mainly addresses the issue of multiple relation semantics. recently, researchers begin to explore the dnn based methods for kge and achieve state-of-the-art results. for example, conve @xcite uses 2d convolution over embeddings and multiple layers of nonlinear features to model kgs. convkb (nguyen et al. , 2018) also use convolutional neural network for kge. conmask (shi and weniger, 2018)uses relationship-dependent content masking, fully residual e1 rel reshape standard conv atrous conv flatten project to embedding space atrous conv e1-des1 e1-desk @xmath3... ... (a) residual e1 rel reshape standard conv atrous conv flatten project to embedding space atrous conv e1-des1 e1-desk @xmath4... ... ... ... ... convolutional neural networks, and semantic averaging to extract relationship-dependent embeddings from the textual features of entities and relations in kgs. more recently, guo et al. (2019) studied the path-level kg embedding learning and proposed recurrent skipping networks(rsns) to remedy the problems of using sequence models to learn relational paths. yao et al. (2020) integrate bert (devlin et al. , 2019) into the kge model. wang et al. (2020) propose coke which uses transformer (vaswani et al. , 2017). vashishth et al. (2020a) extend conve by increasing the interactions with feature permutation, feature reshaping, and circular convolution. most recently, graph based neural network(gnn) methods are achieving more and more attentions. schlichtkrull et al. (2018) propose r-gcn, which is a graph based dnn model that uses neighboring information of each entity. bansal et al. (2019) propose a2n, an attention-based model based on graph neighborhood. shang et al. (2019) propose a weighted graph convolutional network based method that mainly utilizes learnable relational specific scalar weights during gcn aggregation. ye et al. (2019) propose vr-gcn, which is an extention of graph convolutional networks for embedding both nodes and relations. shang et al. (2019) propose sacn that takes the benefit of both gcn and conve. vashishth et al. (2020b) propose compgcn which jointly embeds both nodes and relations in a relational graph. however, as pointed out by @xcite that most of these existing dnn-based or gnn-based kge methods are very complex and time-consuming, which prevents them be used in some on-line or real-time application scenarios. ## acre model we denote our model as acre (the abbreviation of atrous convolution and residual embedding). in this study, we design two structures to integrate the standard convolution and atrous convolutions together. one is a serial structure as shown in figure 1 (a) , and the other is a parallel structure as shown in figure 1 (b ). we will introduce them one by one in the following. ## conclusions in this paper, we propose acre, a simple but effective dnn-based kge model. we make comprehensive comparisons between acre and many state-of-the-art baselines on sis diverse benchmark datasets. extensive experimental results show that acre is very effective and it achieves better results than the compared baselines under most evaluation metrics on six benchmark datasets. the main contributions of our method are summarized as follows. first, to our best knowledge, this is the first work that uses different kinds of convolutions for the kge task. second, we propose two simple but effective learning structures to integrate different kinds of convolutions together. third, the proposed model has much better parameter efficiency than the compared baselines.