M o SE : Modality Split and Ensemble for Multimodal Knowledge Graph Completion (2022)

Abstract: Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share relation representation across modalities. This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality. Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically. Experimental results on three KG datasets show that MoSE outperforms state-of-the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC.

## introduction multimodal knowledge graphs (mkgs) organize multimodal facts in the form of entities and relations, and have been successfully applied to various knowledge-driven tasks @xcite @xcite . to address the inherent incomplete problems in mkgs, multimodal knowledge graph completion (mkgc) has been proposed @xcite , which utilizes auxiliary visual or text information to help predict missing entities. as shown in figure 1a , given the head entity and the relation country -→ , mkgc is required to predict the tail entity £ ¢ ¡ the united states of america . it can be observed that the descriptions attached to entities provide supplementary information for entity prediction. existing mkgc methods usually share a common relation embedding across all modalities for a pair of entities, which tightly couples multiple relations from different modalities. we define this paradigm of mkgc as tight-coupling relation (tcr). as shown in figure 1b , according to the way that the relations from different modalities are coupled, existing methods can be roughly divided into two categories: implicit tcr (i-tcr) meth-ods and explicit tcr (e-tcr) methods. i-tcr methods @xcite usually first fuse multimodal information of an entity into a single embedding, and then learn a unified relation representation based on the embedding. e-tcr methods @xcite @xcite directly model the relationship between separate multimodal information of entities without fusion. they usually learn a single relation embedding to simultaneously represent all intra-modal and inter-modal relations. although existing mkgc methods have achieved promising results, they are limited by tcr in two folds: (1) modality relation contradiction. the tcr usually simultaneously represents multiple relations from different modalities only with a single embedding. however, for a pair of entities, the relation from one modality probably contradicts that from another modality. for example, as shown in figure 1a , the description "american sitcom" of entity the united states of america , while the images do not. the inherent contradiction of tcr results in modality interference during representation learning in mkgs. (2) modality difference ignorance. based on tcr, existing methods usually treat the input in different modalities equally and make a unified prediction, which ignores the difference of modality importance. however, different modalities vary in data quality and entity coverage, and should contribute to the final prediction in varying degrees. to overcome the above limitations, we propose a modality split learning and ensemble inference framework, mose. as shown in figure 1b , in the training phase, mose decouples tcr and learns multiple modality-split relation embeddings instead of a single modality-shared one, which alleviates mutual interference between modalities. in the inference phase, mose first makes predictions for each modality separately based on the modalitysplit embeddings, and then merges them into the final prediction. we explore the best combination of modality predictions with various ensemble methods, and model the modality importance by modulating the modality weights dynamically. experimental results and analysis on three widely-used datasets show that mose outperforms state-of-theart methods for mkgc task. overall, the contributions of this paper can be summarized as follows: existing researches on mkgc mainly focus on extending unimodal knowledge graph embedding (kge) models to further exploit multimodal information. we notice that for a pair of entities in an mkg, existing multimodal kge methods all exploit a modality-shared relation embedding which tightly couples multiple relations from different modalities, which we call tight-coupling relation (tcr). we divide existing methods to two categories: implicit tight-coupling relation (i-tcr) methods and explicit tight-coupling relation (e-tcr) methods. ## conclusion in this paper, we propose a novel modality split learning and ensemble inference framework for multimodal knowledge graph completion called mose. mose first decouples modality-shared relation embedding to modality-split relation embeddings and performs modality-split representation learning in the training phase, aiming at overcoming modality relation contradiction. then, mose exploits three ensemble inference techniques to combine the modality-split predictions by assessing modality importance. experiment results demonstrate that mose outperforms state-of-the-art methods for mkgc task on three widely-used datasets.