Multilingual Knowledge Graph Completion With Joint Relation and Entity Alignment
Abstract
Knowledge Graph Completion (KGC) predicts missing facts in an incomplete Knowledge Graph.
We study Multilingual KGC for KGs that associate entities and relations with surface forms from different languages.
In such a setting, an entity (or relation) may be mentioned as separate IDs in different KGs, necessitating entity alignment (EA) and relation alignment (RA).
In addition to
EA and RA being important subtasks for Multilingual KGC, we posit that high conﬁdence fact predictions may also, in turn, add valuable information for alignment tasks, and vice versa.
In response, we present the novel task of jointly training multilingual KGC, EA and RA models.
Our approach, A LIGN KGC, uses mBERT-based surface form overlap for EA, and combines it with a KGC approach, extended to the multiple KG setting via a loss term that incentivizes RA.
On experiments with DBPedia in ﬁve languages, we ﬁnd that A LIGN KGC achieves up to 17% absolute
MRR improvements in KGC compared to a strong completion model that combines known facts in all languages.
It also outperforms an mBERT-only alignment baseline for EA, underscoring the value of joint training for these tasks.
1. Introduction
A knowledge graph (KG) has nodes representing entities and directed edges representing relations between subject and object entities.
An entity has a unique node (ID).
Relations also have canonical labels such as born-in or works-at, with associated IDs.
A KG is usually associated with a single human language — an entity node (or relation ID) is associated with one or more surface forms in that language.
E.g., the ID for the country USA may have aliases like “United States of America".
KGs are usually very incomplete, as curators struggle to keep up with the real world.
KG completion (KGC) is thus a strongly motivated problem and studies the prediction of true facts unknown to the KG.
While the problem has been intensively researched over the last few years [Bordes et al., 2013, Trouillon et al., 2016, Jain et al., 2018, Sun et al., 2019], most KGC research is applicable to only one KG in one language at a time.
However, different language speakers would maintain separate KGs in their own languages.
Independent completions of each KG may not be optimal, since information from one KG will likely help completion of the other.
Each KG will give a different ID, and often also a different surface form, to the same entity, such as “Estados Unidos de América” for the USA entity in a Spanish KG.
This leads to the problem of
ID proliferation.
Recent work on entity alignment (EA) across KGs attempts to assign a unique ID to all nodes representing the same entity [Chen et al., 2017, Sun et al., 2017, 2018, 2020, Cao et al., 2019, Chen et al., 2021, Tang et al., 2020].
A related but under-explored task is relation alignment (RA) — assigning synonymous relations in different KGs the same relation ID.
We note that RA involves global evidence, because the decision to merge two relations in two languages can have far-reaching consequences to many facts in both KGs.
Our key contribution is to recognize and exploit the synergy between multilingual KGC, EA and RA tasks.
Entity and relation alignments expose a KGC algorithm to more facts, which can lead to better completion.
Conversely, a high-conﬁdence completion can give additional evidence to align its constituent entities and relation.
(See Appendix A for an extended example.)
In this paper, we present A LIGN KGC, a multi-task system that learns to optimize for KGC, EA and RA jointly.
A LIGN KGC uses state-of-the-art mBERT-based models for EA and combines it with an extension of the state-of-the-art ComplEx model [Trouillon et al., 2016] for single KGC.
A LIGN KGC uses a novel subject-object signature of each relation, which it represents as a bag of embeddings, and compares them for equivalence and implication between relations.
A LIGN KGC uses this information as additional term in its loss function for better RA and multilingual KGC performance.
We evaluate A LIGN KGC on slices of DBPedia in ﬁve languages.
We compare it against two single-KGC systems, one trained over each monolingual KG separately, and the other trained over a union of all KG facts.
Compared to these strong baselines, we ﬁnd that A LIGN KGC achieves substantial accuracy boost due to better entity and relation alignment, obtaining up to 17% absolute
MRR improvements in KGC across languages, around 20% absolute HITS@1 gain for RA for rare relations, around 26% absolute HITS@1 gain for RA for more frequent relations, and around 61% absolute HITS@1 gain for EA.
A LIGN KGC also achieves a 5.3% absolute HITS@1 gain over an even stronger EA baseline that uses mBERT embeddings on entity names.
These results underscore the value of joint training for these tasks.
2. Notation and preliminaries
A knowledge graph (KG) provides a graph-view to a knowledge-base (KB).
A KG consists of entities
E and relations (aka relation types) R .
A KG instance is a triple ( s, r, o ) where s, o ∈ E and r ∈ R .
These are all canonical IDs.
Each ID is associated with one or more textual aliases.
KGC task:
For any single KG, training data is provided as ( s, r, o ) triples.
A test instance has the form ( s, r, ?) or (? , r, o ) where the system has to predict o or s .
Multiple correct values are possible.
The evaluation protocol takes a system-ranked list of candidate objects or subjects, and then measures MRR or hits@ K .
Many KG embedding and KGC algorithms have been proposed in the last few years.
ComplEx [Trouillon et al., 2016] with all negative instances (no sampling) is a simple model that gives near state-of-the-art predictions [Jain et al., 2020a]. It has also formed the basis for many KGC extensions (e.g., [Jain et al., 2020b]). So we use it as our baseline KGC gadget.
ComplEx deﬁnes a triple score as f ( s, r, o ) =
(cid:60) ( (cid:104) s , r , o (cid:63) (cid:105) ) , where c (cid:63) is complex conjugate, (cid:104)· · · (cid:105) is a 3-way elementwise inner product and (cid:60) ( · ) is the real part of a complex number.
Using f , ComplEx deﬁnes
Pr( o | s, r ) =
e f ( s,r,o ) (cid:46) (cid:80) o
[Equation]
e f
[Equation]
s
[Equation]
r
[Equation] o [Equation] (1) and the log-likelihood KGC loss as L KGC = (cid:80) ( s,r,o ) ∈ KG − log Pr( o | s, r ) − log Pr( s | o, r ) .
Alignment tasks:
We consider a set L = { l 1 , l 2 , . . .
} of languages.
KG l represents the KG supported by the language l .
An entity in this KG is called e l .
A relation in this KG is called r l .
While trying to improve KGC for target language l (cid:48) , KGs in multiple source languages l may be combined.
An equivalence between entities e l and e l (cid:48) in KGs of two languages l and l (cid:48) is speciﬁed as e l ≡ e l (cid:48) , also written as the triple ( e l , ≡ , e l (cid:48) ) .
Similarly, an equivalence between relations r l and r l (cid:48) is speciﬁed as r l ≡ r l (cid:48) .
Other types of alignment may be possible between relation pairs, such as r l ⇒ r l (cid:48) , which means, for all s, o such that ( s, r l , o ) holds, so does ( s, r l (cid:48) , o ) .
During training, a set of entity equivalences { ( e nl , ≡ , e nl (cid:48) ) : n = 1 , . . . , N } and a set of relation equivalences { ( r ml , ≡ , r ml (cid:48) ) : m = 1 , . . .
, M } are revealed to the system between each KG l and KG l (cid:48) .
The goal of the system is to infer additional entity and relation equivalences.
The system is usually called upon to produce a ranked list of equivalences, which is evaluated using HITS@ K or MRR.
To achieve KGC enhanced with alignment, the system has to infer additional triples in each
KG, making best use of the revealed equivalences, while also inferring additional alignment triples between entities or relations across KGs of different languages.
3. Proposed methods 3.1 KGC baselines
When ComplEx is applied to any one KG in isolation, we call this method KGC MONO .
Perhaps the most straight-forward way to apply a KGC system across multiple KGs is to compute KG U
=
(cid:83) l ∈ L KG l such that for all revealed entity equivalences { ( e l , ≡ , e l (cid:48) ) } , collapse their node pairs, and rename all equivalent r l , r l (cid:48) relation IDs to a common new ID.
A standard KGC system can work on the resulting KG U , and we call this method KGC UNION .
3.2 Joint alignment and completion
For both textual and structured inputs, the problem of inferring a predicate or relation as entailed by another has been well-studied [Lin and Pantel, 2001, Bhagat et al., 2007, Nakashole et al., 2012].
With that work as our point of departure, we ask: How similar are two relations r l , r l (cid:48) in KG U ?
Comparing their embedding vectors r l , r l (cid:48) , as computed by a KGC algorithm such as ComplEx, gives an estimate.
However, when training these embeddings, we found that a simple union of facts (KGC MONO ) did not yield the best results.
Instead, guiding the training through an auxiliary loss, computed using the following explicit relation signatures helped the model learn better embeddings.
Here s, o are interpreted as canonical IDs (hence, ‘hard’).
This will be generalized shortly.
Deﬁnition 1 ( Hard SO-signature).
The (‘hard’) subject-object (SO) signature of a relation is deﬁned as SO( r ) =
{ ( s, o ) : ( s, r, o ) ∈ KG U } .
3.2.1 H ARD SO- OVERLAP & J ACCARD SIMILARITY
Deﬁnition 2 ( SO-overlap).
The (hard) overlap between two relations r l , r l (cid:48) is | SO( r l ) ∩
SO( r l (cid:48) ) | .
Jaccard similarity can then be used as a standard symmetric belief that two relations IDs in different languages are equivalent:
[Equation]
If b J ( r l ⇔ r l (cid:48) ) exceeds a threshold θ (tuned using dev set), we add ( r l , r l (cid:48) ) to set A J of ‘silver’ alignments.
We deﬁne an additional RA loss term encouraging such embeddings to be aligned:
[Equation]
We call this scheme Jaccard .
3.2.2 A SYMMETRIC SUBSUMPTION
A problem with Jaccard similarity is that it can give a symmetric high score to relation pairs having asymmetric implications between them.
E.g., in the DBP5L data set, Jaccard similarity gives a large symmetric similarity score between locationCity and headquarter , or keyPerson and founders relations in different KGs.
However, these relation pairs clearly involve asymmetric implication, and should not be linked.
Therefore, we need an asymmetric belief measure ( b A ) for one relation subsuming another, which we deﬁne as
[Equation]
[Equation]
The asymmetry in (4) refrains from pushing those relation pairs together.
This avoids some erroneous inferences, generally boosting precision — possibly at the expense of some recall, as this may also remove some “reasonable inferences”.
Extending the logical statement ( r l ⇔ r l (cid:48) )
iff ( r l ⇒ r l (cid:48) )
∧ ( r l (cid:48) ⇒ r l ) to the fuzzy domain, we get b A ( r l ⇔ r l (cid:48) )
=
min { b A ( r l ⇒ r l (cid:48) ) , b A ( r l (cid:48) ⇒ r l ) } , which we readily see as equivalent to |
[Equation]
Similar to Eqn.
(3), we incentivize the model to align r l and r l (cid:48) if their b A ( r l ⇔ r l (cid:48) ) is high.
In particular, we search for pairs of ‘silver’ alignments ( r l , r l (cid:48) )
∈
A A , where r l = argmax r (cid:48)
l b A ( r l (cid:48)
⇒ r (cid:48) l ) and r l (cid:48)
= argmax r (cid:48) l (cid:48)
b A ( r l ⇒ r (cid:48) l (cid:48) ) .
In other words, we want to apply the loss only to partner relations with mutually largest implication beliefs in that language.
We deﬁne an asymmetric RA loss as:
[Equation]
We call this method Asymmetric .
3.2.3 S OFT OVERLAP AND APPROXIMATION
In the above deﬁnitions involving SO , we assumed the ( s, o ) pairs were represented using canonical entity IDs.
Unless explicit entity equivalences are provided, ‘hard’ SO overlap will underestimate relation similarities.
We now extend these ideas to redeﬁne SO-signatures using entity embeddings, which can be trained via gradient descent.
Recall that the KGC system obtains embedding vectors e for each entity e in the KG.
Overloading notation, we modify our earlier deﬁnition of subject-object signature.
Deﬁnition 3 ( Soft SO signature).
For each relation r , SO( r ) =
( s , o ) : ( s, r, o ) ∈ KG U } .
Each element is the concatenation of the subject and object embedding vectors .
The i th embedding pair in SO( r ) is denoted SO( r )
[ i ] .
Two such pairs can be compared by, say, extending cosine similarity with an AND-semantic: sim (cid:0) ( s , o ) , ( s (cid:48) , o (cid:48) ) (cid:1) =
σ (cos( s , s (cid:48) ))
σ (cos( o , o (cid:48) )) , where σ is the sigmoid nonlinearity.
This captures the requirement that the subjects have to be similar and the objects have to be similar.
The key challenge is to compare two sets of such SO vectors, SO( r l ) , SO( r l (cid:48) ) for extending Defn.
A simple approach is to use a network to encode each SO-set [Zaheer et al., 2017, Lee et al., 2019] into a set embedding and then compare these (see Appendix B for details).
However, recent work [Pabbaraju and Jain, 2019, Tang et al., 2020] suggests that we should drill down to pairwise interactions between set elements.
To that end, consider two relations from different languages, r l , r l (cid:48) , that are candidates for alignment.
Suppose the ( s , o ) pairs of r l are indexed by i and pairs of r l (cid:48) are indexed by j .
We build a matrix A r l ,r l (cid:48) of pairwise cosine similarities: A r l ,r l (cid:48)
[ i, j ] = sim (cid:0) SO
( r l )[ i ] , SO
( r l (cid:48) )
[ j ] (cid:1) .
Deﬁnition 4 ( Soft SO-Overlap).
The continuous extension of | SO( r l ) ∩
SO( r l (cid:48) ) | , denoted by
SoftOv( r l , r l (cid:48) ) , is deﬁned as the value of the maximal matching on the weighted bipartite graph induced by A r l ,r l (cid:48) .
Note that SoftOv( r l , r l (cid:48) ) depends on entity embeddings, which are to be trained.
Ideally, we should be able to backpropagate various KGC and alignment losses past the solution of the matching problem to the entity and relation embeddings.
Gumbel-Sinkhorn matrix scaling [Cuturi, 2013,
Mena et al., 2018] can be used for this purpose, but it is computationally expensive at KG scales.
(See Appendix C for details.)
Here we use a computationally cheaper approximation: only if i is j ’s strongest partner ( i = argmax i (cid:48)
A [ i (cid:48) , j ] ) and j is i ’s strongest partner ( j = argmax j (cid:48)
A [ i, j (cid:48) ] ), we choose edge ( i, j ) in our matching.
For each such edge ( i, j ) , we add in SoftOv( r l , r l (cid:48) )
a score increment σ (cid:0)
A r l ,r l (cid:48)
[ i, j ] w + c (cid:1) , where σ is the sigmoid nonlinearity, and w > 0 , c ∈ R are model parameters trained along with all embeddings.
We continue to use (4)-(5) as deﬁned for Asymmetric , except we replace the hard overlap | SO( r l ) ∩ SO( r l (cid:48) ) | by SoftOv( r l , r l (cid:48) ) .
We denote the beliefs computed thus by b SA (for soft asymmetric).
The rest of the machinery of the previous section follows, i.e., we compute ‘silver’ alignments ( r l , r l (cid:48) )
∈
A SA , leading to a soft asymmetric RA loss as:
[Equation]
[Equation] We call the resulting method SoftAsymmetric .
3.2.4 A DDING TEXT SIGNALS TO A LIGN KGC
BERT-INT [Tang et al., 2020] demonstrated the power of using mBERT
[Devlin et al., 2019] to derive text-based features that effectively pull together duplicate entities.
A short text passage t ( e ) is collected from the name (alias) and description of each entity e , which is then sent into mBERT, and the [CLS] embedding output by mBERT used as the textual representation of e .
Optionally, to better map textual embeddings into the KG space, we use a small feed-forward network mlp(mbert( t ( e ))) , which, for economy of notation, we will write as mbert( e ) .
For entity alignment, we deﬁne an additional loss term L EA = (cid:80) e l ,e l (cid:48) cos(mbert( e l ) , mbert( e l (cid:48) ))
(cid:107) e l − e l (cid:48) (cid:107) {+
+}
The intuition is that when the mBERT embeddings are similar, we want the L 1 distance between KG embeddings to be small, and vice versa.
Because we found Asymmetric generally better than Jaccard, we add the above mBERT-based loss term L EA to two previous variations. When we add it to Asymmetric, we call the resulting system Asym.+mBERT . When we add it to SoftAsymmetric, the resulting system is called
SoftAsym.+mBERT , which is the full form of A LIGN KGC , with all other variations regarded as ablations.
In a similar way, we can also obtain a textual embedding of a relation and use that to create a textual RA loss.
In our test bed, we did not have reliable surface forms of relations in multiple languages, so we turned off this component.
Further details of this loss are described in Appendix D.
4. Experiments
We study the following research questions through our experiments.
(1) Does our joint KGC, EA and RA model (A LIGN KGC) improve upon independent models for each of these tasks?
(2) Is there any cross-lingual transfer of knowledge, for facts that may be mentioned in one language, but are queried in another?
(3) What are the incremental contributions of soft overlap and mBERT’s surface form encoding in the overall performance of A LIGN KGC?
(4) How does performance vary with the number of seed alignments (both entity and relation) provided at training time?
4.1 Experimental Settings
Dataset: Standard mono-lingual KGC datasets such as FB15k, FB15k-237, WN18, WN18RR, and
YAGO-3-10, are not suited for evaluation of multi-lingual alignment or KGC.
[Table]
A few recent EA datasets such as DBP15K [Sun et al., 2017] and DB2.0 [Sun et al., 2021] provide gold EAs (or ‘dangling’ nonEAs in DB2.0) between two languages at a time. They do not provide direct support for synergistic training and evaluation of RA and KGC at the same time. Adaptations of such benchmarks in either direction appear fraught with potential unfairness.
If we evaluate A LIGN KGC on two languages at a time, it loses its essential advantage of EAs across KGs in multiple languages, together with integrated RA and KGC supervision.
On the other hand, if we collapse all but the target language into a “super language”, we severely distort the distribution of entity equivalence class sizes and frequency of relations, for other systems.
[Figure]
Most suited for our purpose is the recently-released DBP5L benchmark [Chen et al., 2020], which is derived from DBPedia in ﬁve languages: English (En),
Greek (El), Spanish (Es), Japanese (Ja) and French (Fr).
We use the same 60-30-10 splits of the KG triples into train-dev-test folds as in Chen et al.
and combine the train sets of all source languages for training.
Importantly, DBP5L also provides about 40% EAs between language pairs.
We randomly pick half of seed entity alignments for training; the rest are used for testing.
Because DBPedia uses a uniform relation vocabulary that is normalized across all languages, it cannot be directly used to test relation alignment.
To test RA capabilities of our models, we create separate ID spaces for each language.
The original normalized IDs create gold data for alignment.
The models do not get any seed relation alignments, and the whole alignment data is used for testing RA.
Note that no asymmetric gold relation implication r 1 ⇒ r 2 are available, only equivalences r 1 ≡ r 2 .
[Table]
Table 1 lists the statistics of the ﬁve KGs in DBP5L. Unsurprisingly, En is the most wellpopulated, with 5–7 times more relations than other KGs.
Figure 2 shows that a majority of the relation labels have associated string descriptions in only one of ﬁve languages (usually English).
However, relations with monolingual aliases account for only 8% of fact triples.
Meanwhile, almost 80% of fact triples use a relation label that has aliases in all ﬁve languages.
Performance metrics: We evaluate A LIGN KGC on all the three tasks — KGC, EA and RA.
As is common when evaluating a system for KGC, we regard test instances ( s, r, ?)
as a task of ranking o (on the basis of scores computed in equations (1)), with gold o ∗ known.
We report MRR (Mean Reciprocal Rank) and the fraction of queries where o ∗ is recalled within rank 1 and rank 10 (HITS).
The ﬁltered evaluation removes valid train or test tuples ranking above ( s, r, o ∗ ) for scoring purposes.
To evaluate the system for EA, we use the test instance ( e l , ≡ , ? l (cid:48) ) as a task of ranking e l (cid:48) s, using the cosine distance between the entity embeddings of the language pair.
We calculate Hits@1 and
Hits@10 on the resulting rankings.
A similar evaluation is done for RA by ranking r l (cid:48)
s for the query ( r l , ≡ , ?
l (cid:48) ) .
All pairs of languages are considered as l and l (cid:48) for these evaluations.
Algorithms compared: For KGC, our baselines are KGC MONO and KGC UNION , and the recent multilingual KGC method called Knowledge Ensemble (KEnS)
[Chen et al., 2020].
KEnS had strictly more information than A LIGN KGC, since it had access to aligned relation IDs at train time.
We report scores from its best reported setting: KEnS b (RotatE).
The full form of A LIGN KGC (SoftAsym.+mBERT) uses the following loss function:
[Equation]
Here L reg is an L 2 regularization on embeddings.
α, β, γ ≥ 0 are tuned hyperparameters.
We also evaluate other variations described in Section 3.2.
To understand the marginal utility of joint training over pre-trained knowledge in EA, we also compare against an extremely competitive mBERT-only baseline, since it is known that mBERT, by itself, can effectively align entities, given their aliases in different languages
[Tang et al., 2020].
[Table] Since completion is most useful around nascent entities, we use the entity surface form(s) as t ( e ) and not, e.g., its full Wikipedia description or infoboxes.
Training policies and hyperparameters: We ﬁne-tune mBERT on entity surface forms to get 300-dim vectors.
We use Adagrad as the optimizer and ﬁne tune hyper-parameters on a dev set.
We select among the following sets of hyper-parameter values: learning rate from {0.2,0.4,0.6,0.8,1}, α from { 0 . 002 , 0 .
02 , 0 . 2 } , β from { 1 , 2 , 5 , 10 } , and γ from { 1 , 5 , 10 , 50 } .
We ﬁx a batch size of 500.
Our best choices for A LIGN KGC were: learning rate=0.8, α =0.02, β =5, and γ =50.
We choose θ =0.01 for Jaccard.
We use a negative sampling of 2000 instances for each positive triple when training ComplEx.
We sample initial entity and relation embeddings via N (0 , 0 . 05) .
As a form of curriculum, we let relation alignments stabilize over a few iterations and then make the equivalence scores trainable.
4.2 Results: KGC performance
Table 3 reports KGC performance on the ﬁve languages.
Not surprisingly, all models outperform
KGC MONO , since it does not perform multilingual training.
KGC UNION obtains much better scores because of combining different language KGs.
Both baselines are outperformed by all A LIGN KGC variants.
Each successive model enhancement — Jaccard, Asymmetric Model, Soft Asymmetric
Model, and addition of mBERT — achieves progressively better performance.
mBERT also provides similar boost to the ‘hard’ Asymmetric variant, but Asym.+mBERT is beaten by SoftAsym.+mBERT.
To analyze the performance further, we split the test set into two subsets: seen and unseen.
The seen test set refers to those test facts that are already seen at train time, but in a KG of a different language.
I.e., a test fact ( s l , r l , o l ) is in seen test if ∃ ( s l (cid:48) , r l (cid:48) , o l (cid:48) ) ∈ KG l (cid:48) ( train )
s.t.
s l ≡ s l (cid:48) , o l ≡ o l (cid:48) and r l ≡ r l (cid:48) , even though the model may/may not know these alignments.
All other test facts are part of the unseen test set.
Qualitatively, performance on the seen split represents the capacity of the model to memorize known facts in a language and align them to another language; the unseen split gives a true sense of a system’s inference capability, since the fact has likely not been read in any language at train time (except if an entity alignment is missing).
We ﬁrst note that the performance on seen test is generally high, suggesting that the models do a good job of memorizing and aligning facts.
We ﬁnd that Jaccard signﬁcantly helps with seen split, since it is able to learn similar embeddings for two aligned relations, and yields high scores for seen tuple in a different language.
Asymmetric loss helps further as it removes false positives in RA, suffered by (symmetric) Jaccard.
But it has a slight negative effect on unseen test instances, likely because of removal of some reasonable inferences.
Making SO-overlap soft and trainable [Figure] enables more information ﬂow, and better learning of embeddings and implication scores.
This leads to signiﬁcant improvement in both test splits.
Finally, addition of mBERT yields massively better entity alignments, which has an impact on all tasks.
We notice that KEnS performs even worse than KGC UNION , even though it has knowledge of all relation alignments.
We attribute this to (1) a stronger KGC baseline of ComplEx (with high negative sampling), compared to KEnS’s use of RotatE with only one negative sample, and (2) A LIGN KGC’s hard alignment policy of providing aligned entities in one uniﬁed KG with a single embedding, instead of KEnS’s approach of ﬁnding nearest entities in each language KG separately and then creating an ensemble.
4.3 Results: Alignment performance
Table 4 reports the performance of the various models on RA and EA tasks.
We ﬁnd that all A LIGNKGC variants outperform KGC UNION baseline by vast margins.
We notice that asymmetric models perform better for more frequent relations.
Because a much larger fraction of triples (see Figure 2) expresses these highly frequent relations, asymmetric models improve aggregate KGC performance.
Because RA has not been studied on DBP5L to our knowledge, these are the best numbers yet.
We also note that Jaccard and Asymmetric models do not have any learnable parameters to explicitly encourage EA — any EA gains are due to better RA and KGC.
Since those models improve EA performance substantially, we have evidence that other tasks can in turn aid better EA.
The learnable asymmetric model, of course, explicitly incentivizes better alignment, leading to a further improvement
Simply computing mBERT embeddings of entity alias texts beats all A LIGN KGC variations that do not use mBERT, by a large margin.
This is corroborated by Tang et al.
[2020]: mBERT-based EA beats most other baselines reviewed in Section 5.
However, joint training with the combination of mBERT with SoftAsymmetric shows a very signiﬁcant improvement in EA, over and above mBERT.
A more detailed report is in Appendix G.
4.4 Effect of the number of seed alignments
We also study the effect of the number of seed alignments available to the model at train time, on task accuracies.
For this we vary the percentage of seed relation alignments (RA%) and entity alignments (EA%) on x and y axes, and observe KGC and EA HITS@1 performances.
Because of diverse and unknown frequencies in its pretraining corpus, the entity alias embeddings provided by mBERT may be of variable quality.
To better control for the amount of information available to the model as we vary EA%, we use the SoftAsymmetric model for this experiment.
Figure 5(a) shows KGC hits@1 and Figure 5(b) shows EA hits@1 against EA% and RA%.
We observe that (predictably) increasing both EA% and RA% improves KGC performance.
At any RA% value, increase in EA% has a powerful impact on both KGC and EA performances.
Owing to the collapsing of entity nodes, it increases the number of triples that are seen in some language — these are easier to reason with for A LIGN KGC.
Moreover, the model is able to infer many relation IDs as synonymous when more EAs are known, due to the explicit incentivization via asymmetric loss in the loss function — these signiﬁcantly improve downstream KGC performance.
At low EA%, increasing RA% has limited KGC impact.
In this setting, entities in different languages have limited connections, and so the KGs are sparsely connected.
Increasing RA% adds further information on fact similarity across KGs, thus improving EA score, but the numbers remain low, due to the overall difﬁculty of the task. [Figure]
At high EA% values, increasing RA% does not further improve EA performance.
This is because aligned entities (as in the DBP5L data) generally target more frequent relations.
Even at low RA%, this enables adequate alignment of frequent relations, leading further to adequate alignment of the entities attached to such relations.
However, we posit that it still improves alignment of other entities (not in our gold set), leading to a substantial increase in KGC performance.
4.5 Soft overlap approximation quality
We described in Section 3.2.3 our fast approximation for maximal matching, required to evaluate the overlap between SO( r 1 ) and SO( r 2 ) .
In Figure 6 we show a scatter plot of a sample of relation pairs; the x-axis is the true optimal overlap found via the Hungarian matching algorithm, and the y-axis is our approximation.
We can see that the approximation is very close to the optimal.
5. Related work KG Completion:
KGC, through learning embeddings for entities and relations, is a denselypopulated research landscape.
Among the best performers are ComplEx [Trouillon et al., 2016], ConvE [Dettmers et al., 2018], and RotatE [Sun et al., 2019].
Almost all such systems are designed for a single KG, or are agnostic to the language used in entity and relation aliases.
KG alignment:
Interest in KG alignment has grown rapidly of late.
MTransE [Chen et al., 2017] uses TransE [Bordes et al., 2013] on each KG, while penalizing large distances between embeddings of equivalent entities.
BootEA [Sun et al., 2018] is semi-supervised bootstrap EA algorithm.
JAPE [Sun et al., 2017] builds attribute- and network neighborhood-based embeddings of entities and combines them with EA constraints.
MuGNN [Cao et al., 2019] uses a graph neural network (GNN) to ﬁnd node embeddings, which are compared to propose ≡ links.
AliNet [Sun et al., 2020] also uses a GNN, with an attention mechanism over larger node neighborhoods.
BERT-INT [Tang et al., 2020] combines local mBERT features with 1-hop neighborhood interactions.
MultiKE
[Zhang et al., 2019] treats EA and RA at par.
However, they use relation names rather than their SO-signatures.
JEANS [Chen et al., 2021] links KG entities to a text corpus.
It uses TransE for the KG, skip-grams for the text, with additional alignment constraints.
6. Conclusion
We have presented A LIGN KGC, a system that jointly learns to complete multiple monolingual
KGs in different languages (KGC) and align their entities and relations.
KGC and entity alignment are known tasks, but relation alignment, to our knowledge, has never been integrated with them.
A LIGN KGC operates on the KG constructed by taking a union of all monolingual KGs, and extends
KGC models to use novel EA and RA loss terms.
In extensive experiments, A LIGN KGC signiﬁcantly improves KGC accuracy, as well as alignment accuracy, underscoring the value of joint alignment and completion.
Acknowledgments: Soumen Chakrabarti is partly supported by a Jagadish Bose Fellowship, grants from Google and IBM, and a Halepete Family Chair.
Mausam is supported by grants from Huawei,
IBM, Google, Bloomberg and 1MG, a Visvesvaraya faculty award by Govt.
of India and Jai Gupta Chair professorship. We thank the IITD HPC facility for compute resources. Appendix A. Example of KGC, EA and RA synergy A.1 EA helps RA
Suppose a KGC+alignment system is given the equivalences between English (En) and Greek (El) entity names:
• Botafogo_de_Futebol_e_Regatas ≡ Μποταφόγκο ντε Φουτεμπόλ ε Ρεγκάτας (a football club) • Brazil ≡ Βραζιλία
Now suppose the system sees these two triples in En and El KGs, respectively:
En: ( Botafogo_de_Futebol_e_Regatas , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) home_arena , Brazil )
El: ( Μποταφόγκο ντε Φουτεμπόλ ε Ρεγκάτας, (cid:58)(cid:58)(cid:58)(cid:58) σπίτι (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) αρένα, Βραζιλία )
If this pattern occurs often enough, the system can infer the relation alignment “ home_arena ” =
“ σπίτι αρένα ”.
This is an example of inferring RA from EA.
A.2 EA, RA helps KGC
Suppose the system knows that
• Rot-WeissEssen (English) ≡ Ροτ Βάις ΄Εσσεν (Greek) (a football club)
• Essen (English) ≡ ΄Εσσεν (a town in Germany) • ( Rot-WeissEssen , homeArena , Essen ) holds in the English KG
Then the system can potentially infer ( Ροτ Βάις ΄Εσσεν , σπίτι αρένα , ΄Εσσεν ) in the Greek KG.
This is an example of EA and RA helping KGC.
A.3 KGC helps EA
Now suppose the KGC system has already learnt the (soft) inference pattern ([club], home_arena , [city]) and ([city], country , [country])
=
⇒ ([club], home_arena , [country]) where home_arena and country ( Χώρα in Greek) are assumed to be ‘universal’ across languages.
Now if we know that ( ΄Εσσεν , Χώρα , Γερμανία ) holds in the Greek KG, we can apply the above rule to infer ( Ροτ Βάις ΄Εσσεν , σπίτι αρένα , Γερμανία ).
If the corresponding fact was already in the English KG as ( Rot-WeissEssen , homeArena ,
Germany ), Then we can infer the EA Γερμανία ≡ Germany .
Appendix B. Differentiable set representations
Given a ﬁnite set X
= { x i } where x i ∈ X , Zaheer et al.
[2017] showed that any set function f ( X ) that is invariant to permutations of X can be expressed as ρ (cid:0)(cid:80) x
∈ X φ ( x ) (cid:1) for suitable transformations
[Table]
ρ and φ .
In practice, capturing correlations between features of x s requires unreasonably deep/wide networks for ρ and φ .
Pabbaraju and Jain [2019] proposed to present X , after applying adversarial permutations, to an order-sensitive recurrent network which can better capture correlations among x s, while using lower network capacity.
The RNN must reduce variability of its set encoding in the face of these permutations.
Lee et al.
[2019] replaced the RNN with a transformer network.
Appendix C. Matching via matrix scaling
Consider two sets { x i :
i ∈
[ I ] } and { y j : j ∈ [ J ] } with a metric d ij = d ( x i , y j ) .
Let T ij ∈ R + be a transportation matrix , subject marginal constraints to (cid:80) i T ij =
c j for all j and (cid:80) j T ij = r i for all i .
Often, r i = c
j = 1 or r i = 1 /I and
c j = 1 /J .
Our goal is to min T (cid:80) i,j d ij T ij .
While this can be solved via linear programming, it is not clear how to backpropagate losses based on the optimal transportation back to networks that output { x i } , { y j } .
(In our case, these set elements are functions of KG embedding vectors, through SO-sets.)
Cuturi [2013] showed that if the objective is mildly augmented with an entropy term λH ( T ) = − λ (cid:80) i,j T ij log T ij , then the optimal transport can be approximated by iterative row and column scaling of ( d ij ) , which are differentiable operations.
However, this is computationally expensive for large KGs.
Appendix D. Exploiting relation descriptions
Along with entities, relations, too, come with textual aliases in some KGs like WikiData.
If we denote the textual description of relation r using t ( r ) , we can also use mlp(mbert( t ( r ))) as additional intrinsic features of r .
(This MLP is different from the one for entities.)
Similar to entity alignment, if b ( r l ⇔ r l (cid:48) ) is high, then we want mbert( r l ) ≈ mbert( r l (cid:48) ) .
Accordingly, we can assess another loss term
[Equation]
In ongoing work, we are preparing a new data set based on WikiData to evaluate the above extension.
Appendix E. Soft SO-overlap generalizes discrete SO-overlap
Proposition 1.
If entities are represented as 1-hot vectors (i.e., not using a continuous embedding space), then soft overlap in Deﬁnition 4 reduces to discrete overlap in Deﬁnition 2.
Proof sketch: it is easy to verify that with 1-hot vectors, soft overlap is exactly the size of the intersection of the two SO sets.
Appendix F. Other ways to use mBERT signals
Beyond what is presented in the main paper, we explored some other ways to incorporate mBERT outputs, but these did not perform as well as the approach we chose in the end.
One idea was to augment the soft SO signature with vectors output from BERT.
Speciﬁcally, we redeﬁne SO( r ) as
[Equation]
This MLP may be trained as BERT is pinned down, or BERT may also be ﬁne tuned.
This suggests the alternative loss term
[Equation]
We may not want a ﬂat cosine (as matrix A r l ,r l (cid:48) was deﬁned), but introduce some and-like semantics for the match between the vectors with four ﬁelds ( s 1 , s (cid:48) 1 ; o 1 , o (cid:48) 1 ) and ( s 2 , s (cid:48) 2 ; o 2 , o (cid:48) 2 ) .
As one example, we might use
[Equation]
Relative weights balancing e and mbert( e ) may also help.
Appendix G. Detailed EA evaluation for all language pairs
In Table 7 we show EA performance on all pairs of languages, with one as source and other as target.
For the two ‘mBERT’ columns, we ﬁne-tune 10 mBERT models, one for each of (cid:0) 52 (cid:1) language pairs.
For the two ‘mBERT(comb)’ columns, we combine all the language entity pairs to train a single global mBERT model, which gives a beneﬁt on all language pairs.
Adding mBERT(comb)
to
AlignKGC provides structural information to improve EA performance signiﬁcantly in a majority of language pairs, particularly the ones with smaller, sparser KGs.
Appendix H. Standard deviation over random initialization
The table below shows the standard deviation over three runs corresponding to AlignKGC+mBERT.
Usually it is considerably smaller than 1%.
[Table]
Appendix I. Statistical signiﬁcance
The table below shows the p -values for a t-test between KGC UNION and Jaccard.
[Table]
Appendix J. Code and data
Please visit https://www.cse.iitb.ac.in/~soumen/doc/AlignKGC/