Multilingual Knowledge Graph Completion with Self-Supervised Adaptive Graph Alignment (2022)

Abstract: Predicting missing facts in a knowledge graph (KG) is crucial as modern KGs are far from complete. Due to labor-intensive human labeling, this phenomenon deteriorates when handling knowledge represented in various languages. In this paper, we explore multilingual KG completion, which leverages limited seed alignment as a bridge, to embrace the collective knowledge from multiple languages. However, language alignment used in prior works is still not fully exploited: (1) alignment pairs are treated equally to maximally push parallel entities to be close, which ignores KG capacity inconsistency; (2) seed alignment is scarce and new alignment identification is usually in a noisily unsupervised manner. To tackle these issues, we propose a novel self-supervised adaptive graph alignment (SS-AGA) method. Specifically, SS-AGA fuses all KGs as a whole graph by regarding alignment as a new edge type. As such, information propagation and noise influence across KGs can be adaptively controlled via relation-aware attention weights. Meanwhile, SS-AGA features a new pair generator that dynamically captures potential alignment pairs in a self-supervised paradigm. Extensive experiments on both the public multilingual DBPedia KG and newly-created industrial multilingual E-commerce KG empirically demonstrate the effectiveness of SS-AGA

## introduction knowledge graphs (kgs) like freebase @xcite and dbpedia @xcite are essential for various knowledge-driven applications such as question answering @xcite and commonsense reasoning @xcite . a kg contains structured and semantic information among entities and relations, where prior knowledge can be instantiated as factual triples (head entity, relation, tail entity), e.g., (apple inc., founded by, steven jobs). as new facts are continually emerging, modern kgs are still far from being complete due to the high cost of human annotation, which spurs on the knowledge graph completion (kgc) task to automatically predict missing triples to complete the knowledge graph. the kg incompletion circumstance is exacerbated in the multilingual setting, as human annotations are rare and difficult to gather, especially for low-resource languages. unfortunately, most efforts for kgc have been devoted to learning each monolingual kg separately @xcite @xcite @xcite , which usually underperform in low-resource language kgs that suffer from the sparseness @xcite @xcite . in contrast, kgs from multiple languages are not naturally isolated, which usually share some real-world entities and relations. the transferable knowledge can be treated as a bridge to align different kgs, which not only facilitates the knowledge propagation to low-resource kgs but also alleviates costly manual labeling for all languages. in this paper, we explore multilingual kg completion (mkgc) @xcite with limited seed alignment across languages. to mitigate language gaps, some efforts have been initiated on multilingual kg embedding methods, which leverage a kg embedding module (e.g., transe @xcite ) to encode each language-specific kg independently and then employ an alignment loss to force pairs of aligned entities to be close maximally @xcite @xcite . however, such approaches mainly involve two limitations: (1) the kg inconsistency issue among different languages is neglected due to the equal treatment for parallel entities; (2) the scarcity of seed alignment hinders the efficient knowledge transfer across languages. concretely, prior methods treat all alignment pairs equally by forcing all parallel entities to be maximally close to each other @xcite @xcite . this ignores potentially negative effects from the kg inconsistency due to the language diversity. for example, as shown in figure 1 , the support english kg in dbp-5l @xcite has much more enriched knowledge (80k facts) than the greek one (13k facts). in order to complete the query (apple inc., founded by, ?) in the resource-poor japanese kg (28k facts), we can transfer more knowledge from resource-rich english kg through the alignment link of steven jobs than that of the low-data greek. however, if roughly pushing steven jobs to be equally close to that english kg and greek kg, the learned embeddings for steven jobs will be similar even though they have different structures, kg capacity, coverage and quality. as such, it will bring in irrelevant information regarding this query and may cause the model to get the wrong answer. thus, we encourage the model to automatically distinguish the underlying inconsistency and transfer knowledge from suitable support kgs foot_0 for better language-specific kgc performance. one the other hand, seed alignment is critical for cross-lingual transfer @xcite , while acquisition of such parallel entities across languages is costly and often noisy. to mitigate such issue, some recent works @xcite propose to generate new alignment pairs based on the entity embedding similarity dur-ing the training process. the generated new pairs can increase the inter-connectivity between kgs to facilitate knowledge transfer. however, simple usage of correlations between entities without any supervision may increase the noise during training, and inhibit the effectiveness of realistic language alignment in kgs @xcite . motivated by these observations, we propose a self-supervised adaptive graph alignment (ss-aga) framework for mkgc. to tackle the knowledge inconsistency issue, ss-aga regards alignment as a new edge type between parallel entities instead of a loss constrain, which fuses kgs from different languages as a whole graph. based on such unified modeling, we propose a novel gnn encoder with a relation-aware attention mechanism, which aggregates local neighborhood information with learnable attention weights and differs the influence received from multiple alignment pairs for the same entity as shown in figure 1(b ). to alleviate the scarcity of seed alignment, ss-aga exploits a new pair generator that iteratively identifies new alignment pairs in a self-supervised manner. this is achieved by masking some seed alignment in the fused kg before gnn encoding and teaching the generation module to recover them. empirically, ss-aga outperforms popular baselines in both public and industrial datasets. for the public dataset, we use the multilingual dbpedia kg @xcite and for the industrial dataset, we create a multilingual e-commerce product kg called e-pkg. our contributions are as follows: (1) we handle the knowledge inconsistency issue for mkgc by treating entity alignment as a new edge type and introducing a relation-aware attention mechanism to control the knowledge propagation; (2) we propose a new alignment pair generation mechanism with self-supervision to alleviate the scarcity of seed alignment; (3) we constructed a new industriallevel multilingual e-commerce kg dataset; (4) extensive experiments verify the effectiveness of ss-aga in both public and industrial datasets. 2 preliminaries 2.1 knowledge graph completion a knowledge graph @xmath0, r, t ) consists of a set of entities e, relations r, and relational facts @xmath1, r, e t )}, where e h , e @xmath2, and @xmath3. entities and relations are represented by their text descriptions. the gnn '()*+ 𝑔 -(•) e / 0 1 e / 0 2 e 3 0 1 e 4 0 1 𝑒 6 7 1 e / 0 1 𝑟 3 𝑟 4 𝑟 6 𝑟 4 e 3 0 2 e 4 0 2 𝑟 6 𝑟 / e 9 0 2 kg 2 kg 1 e 6 0 2 e / 0 2 𝑟 9 𝑟 / 𝑟 3 𝐺 ;<=> e 3 0 1 e 4 0 1 e 6 0 1 e / 0 1 𝑟 3 𝑟 4 𝑟 6 𝑟 4 e 3 0 2 e 4 0 2 𝑟 6 𝑟 / e 9 0 2 kg 2 kg 1 e 6 0 2 e / 0 2 𝑟 9 𝑟 / 𝑟 3 ? 𝐺 ;<=> (a) seed alignment new alignment train inference relations 𝐺 ;<=> @-=@xmath4;<=> (b) new pair generation gnn '()*+ 𝑔 -(•) gnn c0 𝑔 d (•) iterative masked recover loss 𝒥 f kgc loss 𝒥 g ? 𝐺 ;<=> kg completion task seeks to impute the missing head or tail entity of a triple given the relation and the other entity. without loss of generality, we hereafter discuss the case of predicting missing tails, which we also refer to as a query @xmath5, r, ?e t ). multilingual kg completion (mkgc) utilizes kgs across multiple languages to achieve more accurate kg completion task on each individual kg @xcite . formally, we are given m different language-specific kgs as g 1 , g 2 , • • • , g m , and only limited entity alignment pairs γ g i ↔g j ⊆ {(e i , e j ) : e @xmath6, e @xmath7. we also call γ g i ↔g j the seed alignment pairs to distinguish it from the new or pseudo alignment. each kg g i has their own relation set r i . we denote the union of relation sets from all kgs as a unified relation mkgc is related to but different from the entity alignment (ea) task @xcite . in mkgc, seed alignment is not direct supervision while the auxiliary input features, all used in the training stage for cross-lingual transfer to boost the kgc results. ## discussion and conclusion in this paper, we propose ss-aga for multilingual knowledge graph completion (mkgc). it addresses the knowledge inconsistency issue by fusing all kgs and utilizing a gnn encoder to learn entity embeddings with learnable attention weights that differs the influence from multiple alignment sources. it features a new pair generation conducted in a self-supervised learning manner to tackle the limited seed alignment issue. extensive results on two real-world datasets including a newly-created e-commerce dataset verified the effectiveness of ss-aga. our current approach may fail to fully exploit the benefit of entity and relation texts. in the future, we plan to study more effective ways to combine text data with graph data for better model performance. we are also interested in studying mkgc where there no alignment pairs are given, which is a very practical setting and our current model is not able to deal with.