OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport
Abstract
Multi-modal knowledge graph embeddings (KGE) have caught more and more attention in learning representations of entities and relations for link prediction tasks.
Different from previous uni-modal KGE approaches, multi-modal KGE can leverage expressive knowledge from a wealth of modalities (image, text, etc. ), leading to more comprehensive representations of real-world entities.
However, the critical challenge along this course lies in that the multi-modal embedding spaces are usually heterogeneous.
In this sense, direct fusion will destroy the inherent spatial structure of different modal embeddings.
To overcome this challenge, we revisit multi-modal KGE from a distributional alignment perspective and propose optimal transport knowledge graph embeddings ( OTKGE ).
Speciﬁcally, we model the multi-modal fusion procedure as a transport plan moving different modal embeddings to a uniﬁed space by minimizing the Wasserstein distance between multi-modal distributions.
Theoretically, we show that by minimizing the
Wasserstein distance between the individual modalities and the uniﬁed embedding space,
the ﬁnal results are guaranteed to maintain consistency and comprehensiveness.
Moreover, experimental results on well-established multi-modal knowledge graph completion benchmarks show that our OTKGE achieves state-of-the-art performance.
Introduction
In the past decades, with the emergence and development of numerous knowledge graphs (KGs) [1, 2], a spectrum of related applications has been widely facilitated, e.g. , question answering [3, 4], semantic search [5, 6], and recommendation systems [7, 8].
Due to the incompleteness of real-world knowledge graphs, link prediction becomes an important procedure for constructing knowledge graphs.
To achieve this goal, Knowledge Graph Embedding (KGE) [9, 10, 11, 12, 13] attracts more and more attention.
It can learn the low-dimensional representations of entities and relations and thus predict missing links.
Nowadays, most of the KGE methods [11, 12, 13] mainly focus on the uni-modal knowledge graphs represented with pure symbols or concepts.
However, most data sources we can access exhibit a multi-modal property, where information from text, image, and video coexists and comprehensively describes the underlying target.
In recent years, researchers [14, 15, 16, 17] point out that knowledge extracted from such multi-modal data can signiﬁcantly improve the quality of the learned knowledge graph representations.
To that end, the study of the multi-modal KGE raises a new wave, and a great many meaningful work [18, 16, 19, 15, 20] emerges and achieves remarkable success.
Speciﬁcally, they usually embed the multi-modal knowledge ( e.g., texts, pictures) of the entity into different spaces, then fuse these multi-modal embeddings with the help of operations such as concat or mean .
The resulting embedding is ultimately employed as a uniﬁed representation for the multi-modal entity.
However, since the embeddings of different modalities are usually in various heterogeneous spaces, direct fusion might destroy the intrinsic distribution and thus lead to inconsistent and incomprehensive representation in the uniﬁed space.
Based on this fact, this paper focuses on the following question:
Can we ﬁnd a new KGE model, which is capable of learning uniﬁed representations of multi-modal entities while overcoming the problem of spatial heterogeneity?
To address this issue, we propose a novel multi-modal
KGE method called Optimal Transport Knowledge Graph
Embeddings ( OTKGE ).
Speciﬁcally, we formulate the fusion procedure of multi-modal knowledge as an optimal transport (OT) problem [21].
As shown in Figure 1, OT can move different modal embeddings to a uniﬁed aligned space by an optimal transport plan while overcoming spatial heterogeneity by minimizing the Wasserstein distance between different distributions.
In this way, we can facilitate the information interaction between multimodal embeddings and obtain a uniﬁed representation by the Wasserstein barycenter.
On top of this, we theoretically demonstrate that the divergence between the distribution of individual source modality and that of the uniﬁed space can be bounded by the Wasserstein distance, suggesting that our proposed method can overcome the spatial heterogeneity issue.
Moreover, experimental results demonstrate the superiority of OTKGE on multi-modal knowledge graph completion tasks.
In a nutshell, our contributions are summarized as three-fold:
• For the modal fusion problem in multi-modal KGE
we propose a new fusion approach based on optimal transport, which can effectively tackle the modal spatial heterogeneity by reducing the Wasserstein distance between different modal distributions.
• Theoretically, we provide an upper bound for the target errors from source multi-modal spaces to the uniﬁed space in OT fusion.
It can provide a theoretical guarantee for modal fusion while overcoming spatial heterogeneity.
• Extensive experiments show that OTKGE achieves state-of-the-art performance on two multi-modal knowledge graph completion tasks.
Related Work
The study of knowledge graphs has attracted widespread attention in recent years.
According to the number of used modalities, the previous methods mainly fall into two branches including uni-modal
KGEs and multi-modal KGEs.
2.1 Uni-modal Knowledge Greaph Embeddings
In the past decades, researchers have proposed various embedding approaches to achieve uni-modal
KGE, named translational distance models, semantic matching models, and neural network models.
First of all, translational distance models are derived from the popular method TransE [9], which proposes to model the relation as the translation operation between entities, and adopts the principle head entity + relation = tail entity .
Subsequently, there are many variants of TransE, e.g. , TransH [10], TransR [22] and TransD [23], which model the relation as the projection operation and model the entity as the vectors, respectively.
Despite the practical advantages, they cannot model some relation patterns ( e.g., symmetric, inverse, and composition) [12].
Semantic matching approaches aim to capture and match latent semantics between entities, which models the plausibility of the triple by semantic matching.
For instance, some representative methods are RESCAL [24], DistMult [25], ComplEx [11], and HolE [26].
It is worth mentioning that RotatE [12] models the relation as the rotation in two-dimensional space, thus realizing the goal to model symmetry/antisymmetry, inversion, and composition patterns.
Inspired by RotatE, there are also some variants of RotatE, such as QuatE [13], DualE [27], and R OT H [28], which introduce rotations into other geometric spaces.
In addition to the conventional methods, there are also some models using neural networks to produce
KGE with remarkable effects, such as R-GCN [29], ConvE [30], ConvKB [31], KBGAT [32], and
A2N [33].
However, some of them are usually computationally expensive and may require pre-train embeddings.
2.2 Multi-modal Knowledge Graph Embeddings
Beyond uni-modal KGE methods, recently a great many multi-modal KGE methods have been proposed, which can be divided into neural network methods and translation methods.
Neural network methods utilize neural networks to learn embeddings for structural and multi-modal knowledge.
For instance, MMRFAN [17] and MKRL [14] adopt the uniﬁed space to learn the textual knowledge and visual knowledge of entities and thus realize the multi-modal KGE.
Unfortunately, these models such as MMRFAN are designed for some speciﬁc cases ( e.g., medical knowledge graphs), and they may be not suitable for other general graph structures [20].
Translation methods follow the principle of TransE and model the relation as the translation for multimodal entities.
For instance, DKRL [18] learns the representations of KGs by exploiting the textual and structural information while IKRL [16] learns representations by integrating visual information and structured information.
However, both two models can only use one of the textual information and visual information.
To tackle this problem, [19] proposes to take the sum of sub-energy functions as the overall score function, which aims to utilize the textual information, visual information, and structured KGs simultaneously.
Then TransAE [15] adds a multi-modal autoencoder based on TransE, which can achieve the goal to obtain the embeddings of multi-modal KGE.
Afterward, MMKRL [20] integrates structured knowledge and multi-modal knowledge in a uniﬁed space with the assistance of concat fusion operation.
In a word, these models fuse heterogeneous multi-modal spaces by concat or mean operations, neglecting the differences between distributions.
Therefore, learning embeddings for multi-modal knowledge graphs is still an open problem to be tackled.
Background on Optimal Transport
Before giving an overview of the optimal transport (OT) problem, we ﬁrst introduce its deﬁnition and present a short introduction of the related background.
Speciﬁcally, OT serves as a way to compare two probability distributions, which can provide a transport plan to transfer one point to another.
Optimal Transport Problem [21].
Suppose we have two set of points X
=
( x (1) , · · · , x ( n ) ) ∈ R n and Y =
( y (1) , · · · , y ( m ) )
∈
R m with the Dirac (unit mass) distribution denoted as δ ( · ) .
Our ﬁrst goal is to obtain the empirical probability measures µ and ν for X and Y , respectively.
To achieve this, we let µ = (cid:80) ni
=
α i δ ( x ( i ) ) and ν = (cid:80) mi =1 β
i δ ( y ( i ) ) , where the weight α =
( α 1 , · · · , α n ) and β = ( β 1 , · · · , β m ) can be regarded as the probability simplexes.
Subsequently, we design a ground cost to measure the distance from point x ( i ) to y ( j ) , which is denoted as C ij .
Further, we need to calculate the transport coupling T between µ and ν , and the T ij can measure the joint probability observing x
( i ) and y ( j ) .
Conveniently, it can be obtained in the procedure of solving the following:
OT ( µ, ν, C ) :
= min T (cid:104) T , C (cid:105) , where { T ∈ R ( n × m ) + | T 1 m = α , T T 1 n = β } .
It is worth noting that (cid:104) T , C (cid:105) := tr ( T T C ) is formulated as the Frobenius inner product of matrices.
Wasserstein Distance.
If we formulate the cost function with the p-norm, i.e. , C ij = || x ( i ) , y ( j ) ||
pp , then the p -th Wasserstein distance between two empirical probabilistic measure can be deﬁned as
W p ( µ, ν ) :=
OT ( µ, ν ; || · || pp ) 1 /p [34, 35, 36].
In this paper, we will set p = 1 , 2 for the sake of convenience, respectively.
Methodology
In this section, we introduce a new KGE method, termed Optimal Transport Knowledge Graph
Embeddings ( OTKGE ) for multi-modal knowledge graphs.
First, we present the deﬁnition of the multi-modal KGE problem.
Problem Setup.
Given a multi-modal knowledge graph G =
( E , R , T ) , where E is the set of entities, R is the set of relations, and T = { ( h, r, t ) | h, t ∈ E , r ∈ R} is the set of KG triples.
Here the entity e is equipped with a structural embedding e S , linguistic embedding e I and visual embedding e V .
Speciﬁcally, the uni-modal KGE method only focuses on e S while multi-modal
KGE can leverage multi-modal embeddings.
Representations for Entities and Relations.
An important procedure in the KGE is to perform semantic matching between entities.
To this end, previous work mainly focuses on studying how to model the relation and embedding space for uni-modal KGE.
In this paper, we take a step beyond to leverage the multi-modal knowledge for the representation of entities and relations.
Supposing we have obtained the structural embeddings (by uni-modal KGE or initialization) and multi-modal embeddings (by pretrained models) of the entity in KGs, then here our goal is to learn the uniﬁed embeddings.
Due to the heterogeneity of multi-modal spaces, one can notice that there exists a gap (caused by factors such as distribution discrepancy and dimensional difference) between different multi-modal embeddings.
To address such an issue, as shown in Figure 2, we propose to realize the multi-modal alignment by optimal transport, and then perform multi-modal fusion to obtain a uniﬁed representation.
Multi-modal Alignment.
Considering that multi-modal knowledge is the auxiliary information, we let multi-modal embeddings align with the structural embeddings.
Here we take the modal alignment between linguistic space I and structural embeddings S as an example, and the process between visual space V and structural space S can be achieved similarly.
With the OT alignment strategy, we can model the process of modal alignment as a process of transferring different each dimension of modal embeddings into an alignment space (structural embedding space).
Speciﬁcally, it can be divided into the following steps: 1) Estimate the distribution µ for linguistic embedding E I and the distribution ν for structural embedding E S ; 2) Find a transport coupling T from µ to ν , where T ij gives the probability to transport from the i -th feature dimension of E I to the j -th dimension of E S ; 3) Use T to calculate the barycenter, and then map E I to the uniﬁed space S and form a new embedding ˆ E I .
Following the aforementioned steps, we denote by u = { u i } ni =1 ∈ ∆ n and v = { v j } mj =1 ∈ ∆ m for the probabilistic simplexes of linguistic and structural embeddings, respectively.
Let µ = (cid:80) ni
=1 u i δ E iI , ν = (cid:80) mj
=1 v j δ E jS represent discrete distributions of linguistic and structural embeddings, respectively, where δ is the Dirac function.
For the sake of convenience, we set u i = 1 /m, v i = 1 /n , where m, n are the dimensionality of the embeddings in E I and E S , respectively.
Then we denote the joint distribution (cid:81) ( µ, ν ) : (cid:81) ( µ, ν ) =
{ T ∈ R n × m + | T 1 m = u , T T 1 n = v } , wheredenotes an all-one vector.
Afterward, we compute the OT coupling T between the measures µ and ν , which can be formulated as follows:
[Equation]
where C ij is the cost function that evaluates the distance between E iI and E jS .
Moreover, W ( µ, ν ) can measure the distribution discrepancy between E I and E S .
Multi-modal Fusion.
After obtaining the transport matrix T , the linguistic embedding E I can be transformed into target-aligned embedding ˆ E I with the barycenter-based strategies [34, 35, 36]:
[Equation]
Once this has been done, the transported linguistic embedding ˆ E I is in the same space as structural embedding E S , where ∆ T is an adjustable transport parameter [37].
Subsequently, we can obtain the aligned visual embedding ˆ E V in a similar way.
Then the next step is to fuse the aligned multi-modal embeddings ( ˆ E I and ˆ E V ) with the structural embedding E S utilizing the strategy inspired by [38]:
[Equation]
where E F is the uniﬁed representation; W ( E i , E ) is the W pp distance between the distribution of E i and that of E ; λ i represents the weight.
The whole procedure is summarized in Algorithm 1.
Algorithm 1:
Input: Distributions µ k supported by multi-modal representations E k , and structural distribution ν supported by the structural embeddings E S .
Output: Uniﬁed representation E F
Initialize The size of input representation n k , the size of fused representation m , the adjustable transport parameter ∆ T .
ν
= 1 m /m foreach k ∈ { I, V, S } do µ k = 1 n k /n k C
S [ i, j ] = (cid:107) E k
[ i ] − E S
[ j ] (cid:107) 22 , ∀ i ∈ [ n k ] , j ∈ [ m ] # [ m ] means 1 , · · · , m
T k = OT ( µ k , ν, C S ) (cid:98) E k = diag
(1 / v )( T Tk + ∆ T )
E k
E F = min E 13 (cid:80) i ∈{ I,V,S } λ i W
( ˆ E i , E )
Relational Transformations.
On top of the aforementioned process, we have obtained uniﬁed representations of entities.
Subsequently, we transform the head entity h in the triple ( h, r, t ) with the assistance of r : h relation = [Equation] ⊗ h (4) where h is the embedding of the head entity h , r is the embedding of the relation r , h relation is the embedding of the transformed head entity. Here ⊗ is the transformations imposed to h by r , which
Score Function and Loss.
Having obtained the transformed head entity, the score function formulates as follows:
[Equation]
where t is the embedding of the tail entity t , and d ( · , · ) represents a distance metric ( e.g., inner product or Euclidean distance).
Afterward, to optimize the parameters, we train the model by minimizing the following loss:
[Equation]
where Y label ∈ {− 1 , 1 } denotes the label of the triple ( h, r, t ) .
Here suppose Ω denotes the set of observed triples, then let Ω − =
E × R × E − Ω be the set of unobserved triples.
In the procedure of training, we adopt the negative sampling strategies ( e.g., uniform sampling or Bernoulli sampling [10]).
Noticing that the multi-modal information is already pre-trained in advance, here we update the structural embedding in the training.
This is also consistent with our original intention of using multi-modal information to assist KGE.
Theoretical Analysis
In this section, we will show that the OT-fusion strategy preserves consistency and comprehensiveness from a theoretical perspective.
First, we introduce the target error [39, 40] as an indicator to measure the distribution relationship between the target uniﬁed space and the source space.
Here we ﬁrst give some necessary deﬁnitions.
Suppose X , Z and Y are sets of entities, learned representations, and labels of triples, respectively.
Notice that both structural embeddings and multi-modal embeddings of the entity are always assumed to obey the same true score function f ∗ : Z → Y in multi-modal KGE problems [18, 16, 19, 15, 20].
Considering the true scoring function f ∗ is unknown, one usually chooses a predictor function f from a hypothesis class F for substitution, ∀ f ∈ F , f : Z → Y .
To simplify the notations, here we take the head entity embedding z ∈ Z as the example to analyze while ﬁxing the relation and the tail entity.
The conclusion for the tail entity can be analyzed similarly.
Afterward, noticing that there exists a approximation error between the hypothesis f and the true score function f ∗ under the distribution µ , we measure it here with (cid:15) µ ( f, f ∗ ) =
E z ∈ µ [ | f
( z ) − f ∗ ( z ) | ] , which is termed target error.
For simplicity of notation, we use the shorthand (cid:15) µ ( f ) =
(cid:15) µ ( f, f ∗ ) .
Moreover, we utilize the 1-Wasserstein distance W 1 ( · , · ) to relate the source multi-modal distribution and target distribution.
Above all, the target error in the OT fusion can be derived from the following theorem:
Theorem 1 (Consistency and Comprehensiveness of the OT Fusion )
We denote by µ S , µ I , µ V and µ F for the distributions of structural, linguistic, visual, and fused uniﬁed embeddings.
Assume the hypotheses f, f ∗ ∈ F are all K -Lipschitz continuous for some K .
Then the following statements hold for every hypothesis f, f ∗ ∈ F :
[Equation]
where W 1 ( · , · ) is the 1-Wasserstein distance, (cid:15) ∗ ( f ) is the target error at the corresponding embedding space ∗ .
The proof can refer to Appendix B.
Remark 1 (For Consistency)
In the RHS of the inequality, (cid:15) F ( f ) is the target error of the uniﬁed embedding, which can be optimized implicitly by our training process; max i ∈{ I,V,S } {W 1 ( µ i , µ F ) } is the maximum of the 1-Wasserstein distance between the individual modality and the uniﬁed space, which can be reduced by minimizing the alignment error in Eq.3.
Meanwhile, the LHS represents the worst uni-modal performance.
In this sense, the theorem above suggests that improving the performance on the uniﬁed space can also improve the performance on individual modals .
Hence, the learned embeddings are ensured to be consistent with modal-speciﬁc information.
Remark 2 (For Comprehensiveness)
Similar to the analysis of consistency, the second inequality in the theorem above shows that by minimizing the alignment error as in Eq.3, it is possible to obtain a reasonably uniﬁed embedding such that the post-fusion error can be smaller than any involved uni-modal error .
This suggests that the OT-fusion strategy can integrate complementary information across different modalities efﬁciently.
Experiments 6.1 Experimental Setup and Results
[Table]
Dataset.
In terms of the link prediction task, we conduct the experiments and evaluate OTKGE with two standard competition benchmarks as shown in Table 1.
There includes multi-modal datasets:
WN9-IMG [41] and FB-IMG [19].
M9-IMG dataset is derived from the subset of WN18 [9], which embraces structural knowledge as triples, and multi-modal knowledge including textual messages and visual images.
FB-IMG dataset is derived from the subset of FB15K [9], which includes structural knowledge consisting of triples extracted from Freebase [42], and multi-modal knowledge embracing textual messages and visual images.
Evaluation metrics.
To evaluate our model, we take entities in the knowledge replacing masked entities in the triples, then we rank all candidate triples through the score function.
The evaluation metrics include the hit rate and the mean reciprocal rank.
Speciﬁcally, we use H@n to represent the hit radio with cut-off values n = 1 , 3 , 10 while we use MRR to represent the mean reciprocal rank in the experiments.
Baselines.
We compare our method to some representative models, e.g. , multi-modal KGE methods including IKRL [16], TBKGE [19], TransAE [15] and MMKRL [20]; uni-modal KGE methods including TransE [9], DistMult [25] ComplEx [11], and RotatE [12].
Implementation Details.
In the course of the experiment, we implement OTKGE 2 with PyTorch and conduct experiments with a single GPU.
To be fair, notice that the true triples will get toprank positions when we conduct the evaluation.
In this way, we adopt the ﬁltered setting [9] to make sure that all true triples will be ﬁltered out.
Moreover, we utilize the grid search to ﬁnd the hyper-parameters and we choose the best models from the validation set by using early stopping.
Speciﬁcally, the embedding size k is searched in { 100 , 200 , 400 , 500 } and the learning rate is searched in { 0 . 001 , 0 .
• The structured embeddings are produced from triples in knowledge graphs, without any external multi-modal sources.
To be speciﬁc, uni-modal KGE methods such as TransE [9] and ComplEx [11] can be used to learn structured embeddings.
• The linguistic embeddings of entities are learned by adopting the word2vec [43] technique.
For instance, we learn the linguistic embeddings of FB-IMG dataset by pre-trained word2vec while we use GloVe [44] for the WN9-IMG dataset.
• The visual embeddings of entities are learned by pre-trained VGG [45] models.
To be speciﬁc, visual embeddings are learned by adopting the VGG-m-128CNN [46] model in FB-IMG datasets.
As for the WN9-IMG dataset, we take the VGG19 [45] model to learn visual embeddings.
[Table]
Results on multi-modal datasets.
To demonstrate the superiority of OTKGE, we compare it with other popular models on WN9-IMG and FB-IMG datasets as shown in Table 2.
One can see that
OTKGE achieves the best performance on both datasets.
On one hand, the uni-modal KGE methods such as TransE can only learn the structural knowledge while the rich multi-modal information cannot be used effectively.
Therefore, TransE does not perform as well as OTKGE, which demonstrates the advantage of multi-modal KGE.
On the other hand, the multi-modal KGE methods such as IKRL use concat or mean to fuse information from multiple modalities, which will be constrained by spatial heterogeneity.
Therefore, their experimental performance is weaker than OTKGE, which demonstrates the effectiveness of OT fusion.
6.2 Experiment Analysis
[Figure]
The embedding comparison for OTKGE and other methods.
To show that OTKGE (here we adopt the version of 2-Wasserstein distance as an example) can learn a better-uniﬁed representation, we choose three kinds of semantic (people, place, and event) entities in the FB-IMG dataset and visualize the entity embeddings as shown in Figure 3.
In these sub-ﬁgures, circles in blue (people), green (place), and red (event) represent three kinds of semantic embeddings, respectively.
On one hand, one can see that using only structural knowledge in TransE can not separate these three kinds of embedding well, which demonstrates the necessity of leveraging multi-modal knowledge.
On the other hand, there are still many entities that are not clearly separated in IKRL due to the heterogeneity of modal space.
Different from this, OTKGE can effectively reduce the Wasserstein distance between different distributions, facilitate the in-depth fusion of different modal knowledge, and thereby learn a better-uniﬁed representation.
The ablation study of components in OTKGE.
As shown in Table 3, we conduct the experiments to study the role that different components play in OTKGE (here we adopt the version of 2-Wasserstein distance as an example).
Speciﬁcally, we replace the fusion procedure via optimal transport by the mean and concat operations, which we denote as OTKGE w/ mean and OTKGE w/ concat; We only use the visual knowledge, which we denote as OTKGE w/o ling; we only use the linguistic knowledge, which we denote as OTKGE w/o visual.
We can see that the performance of these ablation versions is reduced compared to the original version of OTKGE, which shows that the OT fusion plays an important role in OTKGE.
[Table]
Conclusion
In this paper
[Table]
we design a new KGE model named OTKGE for multi-modal knowledge graphs, where we formulate the fusion procedure by optimal transport theory.
Previous work usually neglects the heterogeneity of distributions in the multi-modal fusion, which will do harm to the interaction of multi-modal knowledge.
To tackle this problem, we propose to transfer the multi-modal information to a uniﬁed space by optimal transport, and fuse the multi-modal information with the Wasserstein barycenter.
Theoretically, we prove that OTKGE is advantageous with its capability in learning multimodal representation and the target error of the multi-modal embeddings and the uniﬁed embeddings can be bounded.
Moreover, empirical experimental evaluations on multi-modal well-established datasets show that OTKGE can achieve overall state-of-the-art performance.
Acknowledgements
This work was supported in part by the National Key R & D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: U21B2038, 61931008, 61733007, U1936208, 6212200758 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the
Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDB28000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX2021298, and in part by China Postdoctoral Science Foundation under Grant 2022M713101.