Dual Attention Network for Cross-lingual Entity Alignment (2020)

Abstract: Cross-lingual Entity alignment is an essential part of building a knowledge graph, which can help integrate knowledge among different language knowledge graphs. In the real KGs, there exists an imbalance among the information in the same hierarchy of corresponding entities, which results in the heterogeneity of neighborhood structure, making this task challenging. To tackle this problem, we propose a dual attention network for cross-lingual entity alignment (DAEA). Specifically, our dual attention consists of relation-aware graph attention and hierarchical attention. The relation-aware graph attention aims at selectively aggregating multi-hierarchy neighborhood information to alleviate the difference of heterogeneity among counterpart entities. The hierarchical attention adaptively aggregates the low-hierarchy and the high-hierarchy information, which is beneficial to balance the neighborhood information of counterpart entities and distinguish non-counterpart entities with similar structures. Finally, we treat cross-lingual entity alignment as a process of linking prediction. Experimental results on three real-world cross-lingual entity alignment datasets have shown the effectiveness of DAEA.

## introduction in recent years, many large-scale knowledge graphs @xcite @xcite are built to represent and organize the explosive information over the internet. they have been widely used in many fields, such as dialogue system @xcite , machine translation @xcite and medicine @xcite . however, the kgs are usually incomplete, so it is necessary to manually track emerging concepts and dynamically update the knowledge bases, making the whole process quite expensive. fortunately, kgs of different languages are often complementary, which means that many components can be shared. to integrate this complementary knowledge, researches begin to pay attention to cross-lingual entity alignment. cross-lingual entity alignment aims at finding entities with the same semantics in kgs of different languages. various methods have been explored for cross-lingual entity alignment. traditional approaches rely on machine translation or feature engineering @xcite @xcite . the effectiveness of these methods depends largely on the quality of the translation and the nature of the definition. recently, many embedding approaches based on graph neural network (gnn) are proposed for cross-lingual entity alignment @xcite @xcite . these methods first represent the entities and relations in low dimensional spaces and then utilize the powerful encoding ability of graph neural network to learn vector representations for entity or relation. finally, a mapping function is used to align the entities from the source knowledge graph to the target one. however, due to the incompleteness of knowledge graphs and the diversity of knowledge, the structure of kgs in different languages is usually quite different. in the entity alignment task, this difference is mainly reflected in two aspects: the non-isomorphism among the neighborhood structures of counterpart entities and information imbalance among the same hierarchy of counterpart entities. the non-isomorphism among the neighborhood structures refers to the neighborhood of the two entities is inconsistent with each other, especially containing different sets of neighboring entities. figure 1 (a) gives a toy example. in (a), entity pairs (a, a ), (b, b ) and (c, c ) denote three pairs of pre-aligned entities. in kg1, entity b is in the one-hop neighbor set of entity a. however, in kg2, entity b is in two-hop neighbors set of entity a . it suggests that the semantically-related entities can appear at different neighborhood hierarchy, which can easily lead to encoding the same pair of entities into different representations when encoding entities with gnn. the information imbalance among the hierarchy of entities means that the degrees are different at the same hierarchy of the corresponding entity. figure 1 (b) gives a toy example of the degree imbalance. in kg1, the one-hop neighborhood of entity a contains five entities, and the two-hop neighborhood contains four entities. however, in kg2, the one-hop neighborhood and the two-hop neighborhood of entity a each contain only two entities. the reason for this phenomenon may be that the two kgs focus on different contents when they are constructed or due to the lack of some knowledge. it may lead to some problems when encoding entities with gnn. for example, many entities at the center of the knowledge graphs have rich neighbor information, like a, they can integrate the information of the whole graph after limited updates, which will lead to little difference in the final representation of these central entities. in entity alignment, this will make it easier for entities with similar neighborhood structures to match together @xcite . to address the issues above, in this paper, we propose a dual attention network for entity alignment (daea). the dual attention consists of the relation-aware graph attention (r-gat) and hierarchical attention. in r-gat, we extend the graph attention network @xcite by modeling relation in graph attention network and incorporate the translational assumption into the self-attention mechanism that we can model the relationship among head, tail, and relation in the graph attention network. by selectively aggregating multi-level neighborhood information, r-gat can alleviate the nonisomorphic difference between corresponding entities' neighborhood structures. inspired by jumping knowledge network @xcite , in the hierarchical attention module, we use an lstm network to learn attention coefficients. it identifies the most useful neighborhood ranges for each entity and adaptively fuses the information of different hierarchy to balance the neighborhood information of peer entities and distinguish entities with similar structures. in entity alignment methods, we design a new entity alignment method, which uses link prediction directly to accomplish entity alignment. we perform thorough experiments with detailed ablation studies and analyses on three entity alignment datasets, demonstrating the effectiveness of daea. ## problem formulation formally, knowledge graphs consist of a set of entities and a set of relations. knowledge facts are stored as a collection of triples @xcite . we define a kg as @xmath0, r, t ), where e and r represent the sets of entities and relations respectively, t is the set of relational triples. let ## our approaches the model daea proposed for entity alignment combines knowledge embedding and graph neural networks. it consists of three modules: the relation-aware graph attention network module, the lstm-based hierarchical attention module, and the knowledge embedding module. our model framework is shown in figure 2 . ## experiments and results analysis figure 4 shows the statistics of degrees of 20 randomly selected pre-aligned entities at different levels from dbp 15k zh-en . here, the degree of an entity refers to the number of all triplets associated with that entity. from figure 4(a ) and (b), we can see that the degree imbalance of one-hop neighborhood exists only in a few entity pairs. however, the imbalance is obvious in the two-hop neighborhood set. for example, in entity pair z14 , the degree difference between two-hop neighborhood entities is more than 1000. figure 4 (d ) shows that the distribution of degree difference in the two-hop neighborhood roughly follows the long tail distribution. ## conclusion in this paper, we introduce a dual attention mechanism entity alignment model daea, which contains the relation-aware graph attention module and lstm-based hierarchical attention module. in the relationaware graph attention module, we model relation into graph attention network and integrate translation hypothesis into the self-attention mechanism, which can mitigate the non-isomorphic difference among the neighborhood structures of counterpart entities. in the hierarchical attention module, we use the lstm attention mechanism to learn the weight of each output layer of the relation-aware graph attention module, which can balance the neighborhood information of peer entities and distinguish noncounterpart entities with similar structures by learning low-level information and high-level information adaptively. we also design a new alignment method based on link prediction. our experiments on three datasets demonstrate the effectiveness of daea. in future work, we will integrate the semantic information into the model and design the appropriate model based on the completion-based method.