A pril E : Attention with Pseudo Residual Connection for Knowledge Graph Embedding (2020)

Abstract: Knowledge graph embedding maps entities and relations into low-dimensional vector space. However, it is still challenging for many existing methods to model diverse relational patterns, especially symmetric and antisymmetric relations. To address this issue, we propose a novel model, AprilE, which employs triple-level self-attention and pseudo residual connection to model relational patterns. The triple-level self-attention treats head entity, relation, and tail entity as a sequence and captures the dependency within a triple. At the same time the pseudo residual connection retains primitive semantic features. Furthermore, to deal with symmetric and antisymmetric relations, two schemas of score function are designed via a position-adaptive mechanism. Experimental results on public datasets demonstrate that our model can produce expressive knowledge embedding and significantly outperforms most of the state-of-the-art works.

## introduction large scale knowledge graphs such as dbpedia @xcite , freebase @xcite , and yago @xcite , have been shown useful to many applications including natural language understanding @xcite , question answering @xcite , and recommender systems @xcite . knowledge graphs (kgs) @xcite are multirelational graphs containing much factual information in the form of a triple (h, r, t), where h represents the head entity, t represents the tail entity, and r represents the relationship between h and t. although the symbolic form of triples effectively represents structured data, it makes kgs hard to calculate semantic similarity between entities and relations. in this paper, we focus specifically on knowledge graph embedding, which maps entities and relations into low-dimensional vector space while preserving the inherent structures of entities and relations. there are diverse relations in kgs. figure 1 shows symmetric and antisymmetric relational patterns in a kg. barack obama and michelle obama are in a marriage relation, and vice versa, therefore, marriage is a symmetric relation. for another case, barack obama sr. is the father of barack obama but barack obama is not the father of barack obama sr., thus father of is an antisymmetric relation. in addition, there are also different relational categories in figure 1 , which are one-to-one, one-to-many, many-toone, and many-to-many. for example, both barack obama and michelle obama are doctorate of j.d. (juris doctor), which belongs to many-to-one relational category. the alma mater of barack obama are columbia university and harvard university, which belongs to one-to-many relational category. many existing models are effective to process different relational categories. transe @xcite interprets the relation as a translation operation on entities in the low-dimensional space, which is straightforward to capture the one-to-one relational category. however, the representations learned by transe suffer from poor expressiveness of complex relational categories, such as one-to-many, many-toone, many-to-many. to address this issue, transh @xcite , transr @xcite , and transd @xcite extend transe by projecting entities or relations into different vector space, which achieve progress on processing complex relational categories and are helpful in handling symmetric and antisymmetric relation patterns. however, it is still challenging since these models remain difficulties in capturing richer semantic features due to their inadequate expressiveness. it is crucial to preserve different relation patterns, especially the symmetry and the antisymmetry @xcite . several efforts have been made for modelling and inferring symmetric and antisymmetric patterns. distmult @xcite exploits a bilinear diagonal model via element-wise product to capture pairwise interaction between head entity, relation and tail entity. however, distmult can only handle symmetric relations and fail to model antisymmetric relations. complex @xcite represents entities and relations in a complex space thus both symmetric and antisymmetric relations can be learned. however, it causes high memory costs due to the need for high dimensional space. moreover, more neural network architectures have been proposed to learn deep expressive features of triples. conve @xcite and @xcite employ convolutional neural network for kg embedding. capse (nguyen et al., 2019) applies capsule neural network @xcite to model the entries at the same dimension in the entity and relation embeddings. although deep neural networks can capture more expressive features, they are computationally expensive as they usually require pre-trained kg embeddings as input for the neural network. to address above issues, this paper proposes a novel knowledge graph embedding model named aprile (attention with pseudo residual connection embedding) to model relational patterns, especially symmetric relation and antisymmetric relation. aprile consists of a triple-level self-attention and a pseudo residual connection. we employ the triple-level self-attention to capture dependency within a triple by treating the factual triple as a sequence. to retain low-level semantic features, we propose a pseudo residual connection by assigning a new embedding vector to each element within the triple as a pseudo identity of original embedding, which is connected to the output of self-attention. both embeddings construct the final representation of each element. to overcome the limitation of dealing only with symmetrical relations, we propose another schema to process antisymmetric relations, in which the role of each embedding can be switched through position-adaptive mechanism. namely, if an entity is the head of a triple, then the second half of its representation will be utilized in the self-attention layer, and the first half will be treated as the pseudo identity. if the entity is the tail of a triple, then the first half of its representation will become the input of the self-attention layer, and the second half will be the pseudo identity. in this way, triple-level self-attention is sensitive to position so that the model can preserve both symmetric and antisymmetric relational patterns. moreover, two schemas of score function are designed to facilitate the position-adaptive mechanism. we conduct extensive experiments on real-world public datasets. the link prediction results on fb15k, wn18, fb15k-237, and wn18rr show that aprile outperforms most of the state-of-the-art kg embedding models. furthermore, the experimental results of each relation on the wn18 dataset indicate that aprile achieves significant improvement than baselines in symmetric and antisymmetric relations. moreover, further experiments on fb15k by relational category illuminate aprile is effective to handle different triple-level self-attention h r t sd ft pseudo residual connection h o r o t o distance h o r o t o translation figure 2: model architecture of aprile. relational categories. in summary, the main contributions of this paper are three-fold: (1) to learn sufficient semantic features and capture the interdependency of h, r, t whin a triple, we propose a novel model aprile that employs the well-designed triple-level attention and pseudo residual connection mechanisms. (2) two schemas of score function are designed based on the combination of different embedding partitions to deal with symmetric and antisymmetric relations. (3) extensive experiments on public datasets demonstrate that aprile can effectively process not only symmetric/antisymmetric relational patterns but also different relation categories, and it significantly outperforms previous state-of-the-art works. ## related work knowledge graph embedding is a critical research issue of kg and a variety of embedding methods have been proposed. translation-based models all follow the translation principle h + r ≈ t. transe @xcite ) represents relations as translations of the entities from head to tail. despite its simplicity and efficiency, transe is overwhelmed when dealing with complex one-to-many, many-to-one and many-to-many relationships @xcite . to overcome the disadvantages of transe, transh @xcite , transr @xcite , transd @xcite introduce projection vectors or matrices to map entities embeddings to different relation vector spaces. bilinear embedding methods utilize product-based scoring functions to match the potential semantics of entities and relations contained in the vector space representation. rescal @xcite captures the interaction between the head entity and the tail entity through the relationship-related matrix. dismult @xcite simplifies rescal for multi-relational representation learning by restricting the relationship-related matrix to a diagonal matrix. although the number of parameters is greatly reduced, it can not handle the antisymmetric relations in general kgs. to model symmetric and antisymmetric relations, complex @xcite firstly introduces complex space to model triple. rotate @xcite takes each relation as a rotation from head entity to tail entity in complex space. recently, cnn-based models have been proposed to learn deep expressive features. conve @xcite uses 2d convolution over embedding and multiple layers of nonlinear features to model the interactions between entities and relationships, of which the head entity and relation are reshaped into a 2d matrix. convkb @xcite adopts cnn to explore the global relationships among same dimensional entries of entities and relational embedding and generalize the transitional characteristics in the transition-based models. our proposed model aprile belongs to the translational embedding methods. more specially, aprile employs triple-level self-attention and pseudo residual connection, which aims to model the relational patterns including symmetric and antisymmetric relations while it can also model different relational categories including one-to-one, one-to-many, many-to-one and many-to-many. ## conclusion and future work in this paper, we have proposed a novel model aprile for knowledge graph embedding. the welldesigned triple-level self-attention and pseudo residual connection enable aprile to model symmetric and antisymmetric relations effectively while it can also deal with the complex one-to-many, many-toone and many-to-many relational categories. moreover, extensive experiments on public benchmark datasets show that aprile outperforms most state-of-the-art baselines. in the future, we plan to extend our method on complex space embedding and plan to handle more relational patterns, thereby providing a deeper insight analysis of the proposed model.