Original: M ULTIMODAL A NALOGICAL R EASONING OVER K NOWLEDGE G RAPHS
Revised: M ULTIMODAL A NALOGICAL R EASONING OVER K NOWLEDGE G RAPHS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Analogical reasoning is fundamental to human cognition and holds an important place in various fields.
Revised: Analogical reasoning is fundamental to human cognition and holds an important place in various fields.

Original: However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge.
Revised: However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge.

Original: Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources.
Revised: Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources.

Original: To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge.
Revised: To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge.

Original: Specifically, we construct a M ultimodal A nalogical R easoning data S et ( MARS ) and a multimodal knowledge graph MarKG .
Revised: Specifically, we construct a M ultimodal A nalogical R easoning data S et ( MARS ) and a multimodal knowledge graph MarKG .

Original: We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task.
Revised: We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task.

Original: We further propose a novel model-agnostic M ultimodal a nalogical r easoning framework with T ransformer ( MarT ) motivated by the structure mapping theory, which can obtain better performance.
Revised: We further propose a novel model-agnostic M ultimodal a nalogical r easoning framework with T ransformer ( MarT ) motivated by the structure mapping theory, which can obtain better performance.

Original: We hope our work can deliver benefits and inspire future research.
Revised: We hope our work can deliver benefits and inspire future research 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Analogical reasoning – the ability to perceive and use relational similarity between two situations or events – holds an important place in human cognition (Johnson-Laird, 2006; Wu et al., 2020;
Revised: Analogical reasoning – the ability to perceive and use relational similarity between two situations or events – holds an important place in human cognition (Johnson-Laird, 2006; Wu et al., 2020;

Original: Bengio et al., 2021; Chen et al., 2022a) and can provide back-end support for various fields such as education (Thagard, 1992), creativity (Goel, 1997), thus appealing to the AI community.
Revised: Bengio et al., 2021; Chen et al., 2022a) and can provide back-end support for various fields such as education (Thagard, 1992), creativity (Goel, 1997), thus appealing to the AI community.

Original: Early,
Revised: Early,

Original: Mikolov et al.
Revised: Mikolov et al.

Original: (2013b)
Revised: (2013b)

Original: ; Gladkova et al. (2016a); Ethayarajh et al.
Revised: ; Gladkova et al. (2016a); Ethayarajh et al.

Original: (2019a)
Revised: (2019a)

Original: propose visual analogical reasoning aiming at lifting machine intelligence in Computer Vision (CV) by associating vision with relational, structural, and analogical reasoning.
Revised: propose visual analogical reasoning aiming at lifting machine intelligence in Computer Vision (CV) by associating vision with relational, structural, and analogical reasoning.

Original: Meanwhile, researchers of Natural Language
Revised: Meanwhile, researchers of Natural Language

Original: Processing (NLP) hold the connectionist assumption (Gentner, 1983) of linear analogy (Ethayarajh et al., 2019b); for example, the relation between two words can be inferred through vector arithmetic of word embeddings.
Revised: Processing (NLP) hold the connectionist assumption (Gentner, 1983) of linear analogy (Ethayarajh et al., 2019b); for example, the relation between two words can be inferred through vector arithmetic of word embeddings.

Original: However, it is still an open question whether artificial neural networks are also capable of recognizing analogies among different modalities.
Revised: However, it is still an open question whether artificial neural networks are also capable of recognizing analogies among different modalities.

Original: Note that humans can quickly acquire new abilities based on finding a common relational system between two exemplars, situations, or domains.
Revised: Note that humans can quickly acquire new abilities based on finding a common relational system between two exemplars, situations, or domains.

Original: Based on Mayer’s Cognitive Theory of multimedia learning (Hegarty & Just, 1993; Mayer, 2002), human learners often perform better on tests with analogy when they have learned from multimodal sources than single-modal sources.
Revised: Based on Mayer’s Cognitive Theory of multimedia learning (Hegarty & Just, 1993; Mayer, 2002), human learners often perform better on tests with analogy when they have learned from multimodal sources than single-modal sources.

Original: Evolving from recognizing single-modal analogies to exploring multimodal reasoning for neural models, we emphasize the importance of a new kind of analogical reasoning task with Knowledge Graphs (KGs).
Revised: Evolving from recognizing single-modal analogies to exploring multimodal reasoning for neural models, we emphasize the importance of a new kind of analogical reasoning task with Knowledge Graphs (KGs).

Original: In this paper, we introduce the task of multimodal analogical reasoning over knowledge graphs to fill this blank.
Revised: In this paper, we introduce the task of multimodal analogical reasoning over knowledge graphs to fill this blank.

Original: Unlike the previous multiple-choice QA setting, we directly predict the analogical target and formulate the task as link prediction without explicitly providing relations .
Revised: Unlike the previous multiple-choice QA setting, we directly predict the analogical target and formulate the task as link prediction without explicitly providing relations .

Original: Specifically, the task can be formalized as ( e h , e t ) : ( e q , ?)
Revised: Specifically, the task can be formalized as ( e h , e t ) : ( e q , ?)

Original: with the help of background multimodal knowledge graph
Revised: with the help of background multimodal knowledge graph

Original: G , in which e h , e t or e q have different modalities.
Revised: G , in which e h , e t or e q have different modalities.

Original: We collect a M ultimodal A nalogical R easoning data S et ( MARS ) and a multimodal knowledge graph MarKG to support this task.
Revised: We collect a M ultimodal A nalogical R easoning data S et ( MARS ) and a multimodal knowledge graph MarKG to support this task.

Original: These data are collected and annotated from seed entities and relations in E-KAR (Chen et al., 2022a) and
Revised: These data are collected and annotated from seed entities and relations in E-KAR (Chen et al., 2022a) and

Original: BATs (Gladkova et al., 2016a), with linked external entities in Wikidata and images from Laion-5B (Schuhmann et al., 2021).
Revised: BATs (Gladkova et al., 2016a), with linked external entities in Wikidata and images from Laion-5B (Schuhmann et al., 2021).

Original: To evaluate the multimodal analogical reasoning process, we follow the guidelines from psychological theories and conduct comprehensive experiments on MARS with multimodal knowledge graph embedding baselines and multimodal pre-trained Transformer baselines.
Revised: To evaluate the multimodal analogical reasoning process, we follow the guidelines from psychological theories and conduct comprehensive experiments on MARS with multimodal knowledge graph embedding baselines and multimodal pre-trained Transformer baselines.

Original: We further propose a novel
Revised: We further propose a novel

Original: M ultimodal a nalogical r easoning framework with T ransformer, namely MarT , which is readily pluggable into any multimodal pre-trained Transformer models and can yield better performance.
Revised: M ultimodal a nalogical r easoning framework with T ransformer, namely MarT , which is readily pluggable into any multimodal pre-trained Transformer models and can yield better performance.

Original: To summarize, our contributions are three-fold: (1) We advance the traditional setting of analogy learning by introducing a new multimodal analogical reasoning task.
Revised: To summarize, our contributions are three-fold: (1) We advance the traditional setting of analogy learning by introducing a new multimodal analogical reasoning task.

Original: Our work may open up new avenues for improving analogical reasoning through multimodal resources.
Revised: Our work may open up new avenues for improving analogical reasoning through multimodal resources.

Original: (2) We collect and build a dataset MARS with a multimodal knowledge graph MarKG, which can be served as a scaffold for investigating the multimodal analogy reasoning ability of neural networks.
Revised: (2) We collect and build a dataset MARS with a multimodal knowledge graph MarKG, which can be served as a scaffold for investigating the multimodal analogy reasoning ability of neural networks.

Original: (3) We report the performance of various multimodal knowledge graph embedding, multimodal pre-trained Transformer baselines, and our proposed framework MarT. We further discuss the potential of this task and hope it facilitates future research on zero-shot learning and domain generalization in both CV and NLP.
Revised: (3) We report the performance of various multimodal knowledge graph embedding, multimodal pre-trained Transformer baselines, and our proposed framework MarT. We further discuss the potential of this task and hope it facilitates future research on zero-shot learning and domain generalization in both CV and NLP.

Original:  B ACKGROUND 2. A NALOGICAL R EASONING IN P SYCHOLOGICAL
Revised:  B ACKGROUND 2. A NALOGICAL R EASONING IN P SYCHOLOGICAL

Original: To better understand analogical reasoning, we introduce some crucial theories from cognitive psychology, which we take as guidelines for designing the multimodal analogical reasoning task.
Revised: To better understand analogical reasoning, we introduce some crucial theories from cognitive psychology, which we take as guidelines for designing the multimodal analogical reasoning task.

Original: Structure Mapping Theory (SMT) (Gentner, 1983) .
Revised: Structure Mapping Theory (SMT) (Gentner, 1983) .

Original: SMT is a theory that takes a fundamental position in analogical reasoning.
Revised: SMT is a theory that takes a fundamental position in analogical reasoning.

Original: Specifically, SMT emphasizes that humans conduct analogical reasoning depending on the shared relations structure rather than the superficial attributes of domains and distinguishes analogical reasoning with literal similarity.
Revised: Specifically, SMT emphasizes that humans conduct analogical reasoning depending on the shared relations structure rather than the superficial attributes of domains and distinguishes analogical reasoning with literal similarity.

Original: Minnameier (2010) further develops the inferential process of analogy into three steps: abduction, mapping and induction, which inspires us to design benchmark baselines for multimodal analogical reasoning.
Revised: Minnameier (2010) further develops the inferential process of analogy into three steps: abduction, mapping and induction, which inspires us to design benchmark baselines for multimodal analogical reasoning.

Original: Mayer’s Cognitive Theory (Hegarty & Just, 1993; Mayer, 2002) .
Revised: Mayer’s Cognitive Theory (Hegarty & Just, 1993; Mayer, 2002) .

Original: Humans live in a multi-source heterogeneous world and spontaneously engage in analogical reasoning to make sense of unfamiliar situations in everyday life (Vamvakoussi, 2019).
Revised: Humans live in a multi-source heterogeneous world and spontaneously engage in analogical reasoning to make sense of unfamiliar situations in everyday life (Vamvakoussi, 2019).

Original: Mayer’s Cognitive Theory shows that human learners often perform better on tests of recall and transfer when they have learned from multimodal sources than single-modal sources.
Revised: Mayer’s Cognitive Theory shows that human learners often perform better on tests of recall and transfer when they have learned from multimodal sources than single-modal sources.

Original: However, relatively little attention has been paid to multimodal analogical reasoning, and it is still unknown whether neural network models have the ability of multimodal analogical reasoning.
Revised: However, relatively little attention has been paid to multimodal analogical reasoning, and it is still unknown whether neural network models have the ability of multimodal analogical reasoning.

Original: 2. A NALOGICAL R EASONING IN CV AND NLP
Revised: 2. A NALOGICAL R EASONING IN CV AND NLP

Original: Visual Analogical Reasoning.
Revised: Visual Analogical Reasoning.

Original: Analogical reasoning in CV aims at lifting machine intelligence by associating vision with relational, structural, and analogical reasoning (Johnson et al., 2017; Prade & Richard, 2021;
Revised: Analogical reasoning in CV aims at lifting machine intelligence by associating vision with relational, structural, and analogical reasoning (Johnson et al., 2017; Prade & Richard, 2021;

Original: Hu et al., 2021; Malkinski & Mandziuk, 2022).
Revised: Hu et al., 2021; Malkinski & Mandziuk, 2022).

Original: Some datasets built in the context of Raven’s Progressive Matrices (RPM) are constructed, including PGM (Santoro et al., 2018) and
Revised: Some datasets built in the context of Raven’s Progressive Matrices (RPM) are constructed, including PGM (Santoro et al., 2018) and

Original: RAVEN (Zhang et al., 2019).
Revised: RAVEN (Zhang et al., 2019).

Original: Meanwhile, Hill et al.
Revised: Meanwhile, Hill et al.

Original: (2019) demonstrates that incorporating structural differences with structure mapping in analogical visual reasoning benefits the machine learning models.
Revised: (2019) demonstrates that incorporating structural differences with structure mapping in analogical visual reasoning benefits the machine learning models.

Original: Hayes & Kanan (2021) investigates online continual analogical reasoning and demonstrates the importance of the selective replay strategy.
Revised: Hayes & Kanan (2021) investigates online continual analogical reasoning and demonstrates the importance of the selective replay strategy.

Original: However, these aforementioned works still focus on analogy reasoning among visual objects while ignoring the role of complex texts.
Revised: However, these aforementioned works still focus on analogy reasoning among visual objects while ignoring the role of complex texts.

Original: Natural Language Analogical Reasoning.
Revised: Natural Language Analogical Reasoning.

Original: In the NLP area, early attempts devote to word analogy recognition (Mikolov et al., 2013b; Gladkova et al., 2016a;
Revised: In the NLP area, early attempts devote to word analogy recognition (Mikolov et al., 2013b; Gladkova et al., 2016a;

Original: Jurgens et al., 2012; Ethayarajh et al., 2019a; Gladkova et al., 2016b) which can often be effectively solved by vector arithmetic for neural word embeddings Word2Vec (Mikolov et al., 2013a) and Glove (Pennington et al., 2014).
Revised: Jurgens et al., 2012; Ethayarajh et al., 2019a; Gladkova et al., 2016b) which can often be effectively solved by vector arithmetic for neural word embeddings Word2Vec (Mikolov et al., 2013a) and Glove (Pennington et al., 2014).

Original: Recent studies have also evaluated on the pre-trained language models (Devlin et al., 2019; Brown et al.,2020; Ushio et al., 2021). However, word analogies mainly measure the quality of word representations and do not explore the analogical reasoning ability of models. Thus, Chen et al. (2022a) builds a knowledge-intensive benchmark to evaluate the analogical reasoning ability of neural models.
Revised: Recent studies have also evaluated on the pre-trained language models (Devlin et al., 2019; Brown et al.,

Original: Nevertheless, Chen et al.

Original: (2022a) mainly focuses on reasoning in the textual domain and does not consider using external knowledge graphs.

Original: In this work, we take the first step to investigate multimodal analogical reasoning over knowledge graphs.

Original: (Luogeng Hua)
Revised: (Luogeng Hua)

Original: Mathematician
Revised: Mathematician

Original: Physicist (Albert Einstein) occupation ?
Revised: Physicist (Albert Einstein) occupation ?

Original: occupation same relation (Luogeng Hua) instance of subclass of (Albert Einstein) instance of
Revised: occupation same relation (Luogeng Hua) instance of subclass of (Albert Einstein) instance of

Original: Man
Revised: Man

Original: Scientist
Revised: Scientist

Original: Physicist subclass of Nikola Tesla occupation instance of occupation instance of (Gauss)
Revised: Physicist subclass of Nikola Tesla occupation instance of occupation instance of (Gauss)

Original: Mathematician (Young Einstein) (Albert Einstein) (Young Einstein)
Revised: Mathematician (Young Einstein) (Albert Einstein) (Young Einstein)

Original: Nikola Tesla ?
Revised: Nikola Tesla ?

Original: Young Tesla identical to identical to Young Tesla
Revised: Young Tesla identical to identical to Young Tesla

Original: Mathematician
Revised: Mathematician

Original: Physicist (Luogeng Hua) occupationcorrespond to identical to correspond to correspond to
Revised: Physicist (Luogeng Hua) occupationcorrespond to identical to correspond to correspond to

Original: Gauss ?
Revised: Gauss ?

Original: Multimodal Knowledge Graph
Revised: Multimodal Knowledge Graph

Original: Single Analogical Reasoning Blended Analogical Reasoning
Revised: Single Analogical Reasoning Blended Analogical Reasoning

Revised: ; Ushio et al., 2021).

Revised: However, word analogies mainly measure the quality of word representations and do not explore the analogical reasoning ability of models.

Revised: Thus, Chen et al.

Revised: (2022a) builds a knowledge-intensive benchmark to evaluate the analogical reasoning ability of neural models.

Original:   T HE M ULTIMODAL A NALOGICAL R EASONING T ASK 3. T ASK D EFINITION
Revised: Nevertheless, Chen et al. (2022a) mainly focuses on reasoning in the textual domain and does not consider using external knowledge graphs. In this work, we take the first step to investigate multimodal analogical reasoning over knowledge graphs.  T HE M ULTIMODAL A NALOGICAL R EASONING T ASK 3. T ASK D EFINITION

Original: In this section, we introduce the task of Multimodal Analogical Reasoning that can be formulated as link prediction without explicitly providing relations .
Revised: In this section, we introduce the task of Multimodal Analogical Reasoning that can be formulated as link prediction without explicitly providing relations .

Original: As shown in Figure 1, given an analogy example ( e h , e t ) and a question-answer entity pair ( e q , ?)
Revised: As shown in Figure 1, given an analogy example ( e h , e t ) and a question-answer entity pair ( e q , ?)

Original: where e h , e t , e q ∈ E a and E a ∈ E , the goal of analogical reasoning is to predict the missing entity e a ∈ E a .
Revised: where e h , e t , e q ∈ E a and E a ∈ E , the goal of analogical reasoning is to predict the missing entity e a ∈ E a .

Original: Moreover, multimodal analogical reasoning is based on background multimodal knowledge graph G =
Revised: Moreover, multimodal analogical reasoning is based on background multimodal knowledge graph G =

Original: ( E , R , I , T ) , where E and
Revised: ( E , R , I , T ) , where E and

Original: R are sets of entities and relations, I and T represent images and textual descriptions of entities.
Revised: R are sets of entities and relations, I and T represent images and textual descriptions of entities.

Original: Note that the relations of ( e h , e t ) and ( e q , e a ) are identical but unavailable, and the relation structure can be analogized implicitly from source domain to target domain without knowing the relations.
Revised: Note that the relations of ( e h , e t ) and ( e q , e a ) are identical but unavailable, and the relation structure can be analogized implicitly from source domain to target domain without knowing the relations.

Original: Specifically, the task can be formalized as ( e h , e t ) : ( e q , ?) , further divided into Single Analogical
Revised: Specifically, the task can be formalized as ( e h , e t ) : ( e q , ?) , further divided into Single Analogical

Original: Reasoning and Blended Analogical Reasoning according to different modalities of e h , e t , e q and e a .
Revised: Reasoning and Blended Analogical Reasoning according to different modalities of e h , e t , e q and e a .

Original: Single Analogical Reasoning.
Revised: Single Analogical Reasoning.

Original: In this setting, the analogy example and the question-answer entity pair involve only one modality.
Revised: In this setting, the analogy example and the question-answer entity pair involve only one modality.

Original: As shown in the middle column of Figure 1, the modalities of the analogy example ( e h , e t ) are identical and opposite to the analogy question-answer pair ( e q , e a ) .
Revised: As shown in the middle column of Figure 1, the modalities of the analogy example ( e h , e t ) are identical and opposite to the analogy question-answer pair ( e q , e a ) .

Original: Based on both visual and textual modalities, this setting can be further divided into ( I h , I t ) :
Revised: Based on both visual and textual modalities, this setting can be further divided into ( I h , I t ) :

Original: ( T q , ?)
Revised: ( T q , ?)

Original: and ( T h , T t ) :
Revised: and ( T h , T t ) :

Original: ( I q , ?)
Revised: ( I q , ?)

Original: where I h , T h represent the modality of e h is visual or textual respectively.
Revised: where I h , T h represent the modality of e h is visual or textual respectively.

Original: Blended Analogical Reasoning.
Revised: Blended Analogical Reasoning.

Original: In the setting, the modality of analogy example ( e h , e t ) are unidentical, which is similar to real-world human cognition and perception 2 .
Revised: In the setting, the modality of analogy example ( e h , e t ) are unidentical, which is similar to real-world human cognition and perception 2 .

Original: Note that Mayer’s theory indicates that humans can have powerful transfer and knowledge recall abilities in multimodal scenarios.
Revised: Note that Mayer’s theory indicates that humans can have powerful transfer and knowledge recall abilities in multimodal scenarios.

Original: Inspired by this, we propose the blended analogical reasoning that can be formalized as ( I h , T t ) :
Revised: Inspired by this, we propose the blended analogical reasoning that can be formalized as ( I h , T t ) :

Original: ( I q , ?) , which means the modalities between e h ( e q ) and e t ( e a ) are different.
Revised: ( I q , ?) , which means the modalities between e h ( e q ) and e t ( e a ) are different.

Original: 3. D ATA C OLLECTION AND P REPROCESSING
Revised: 3. D ATA C OLLECTION AND P REPROCESSING

Original: We briefly introduce the construction process of the dataset in Figure 2.
Revised: We briefly introduce the construction process of the dataset in Figure 2.

Original: Firstly, we collect a multimodal knowledge graph dataset MarKG and a multimodal analogical reasoning dataset MARS, which are developed from seed entities and relations in E-KAR (Chen et al., 2022a) and BATs (Gladkova et al., 2016a).

Original: Secondly, we link these seed entities into the free and open knowledge base Wikidata 3 for formalization and normalization.

Original: Thirdly, to acquire the image data, we further search from the Google engine and query from the multimodal data Laion-5B (Schuhmann et al., 2021) by the text descriptions of entities.

Original: Then, an image validation strategy is applied to filter low-quality images.

Original: Lastly, we sample high-quality analogy data to construct MARS. A detailed description of the data collection and processing to create our datasets are Appendix B.1 and B.
Revised: Firstly, we collect a multimodal knowledge graph dataset MarKG and a multimodal analogical reasoning dataset MARS,

Original: MediaAPI
Revised: MediaAPI

Original: Laion-5B
Revised: Laion-5B

Original: Luogeng Hua mathematician occupation instance of
Revised: Luogeng Hua mathematician occupation instance of

Original: Luogeng Hua mathematician man scientist
Revised: Luogeng Hua mathematician man scientist

Original: Google
Revised: Google

Original: Gromov Gromov
Revised: Gromov Gromov

Original: Luogeng Hua
Revised: Luogeng Hua

Original: Filter
Revised: Filter

Original: Step1: Collect Analogical Entities and Relations
Revised: Step1: Collect Analogical Entities and Relations

Original: Step2: Link to Wikidata and Retrieve Neighbors
Revised: Step2: Link to Wikidata and Retrieve Neighbors

Original: Step3: Acquire and Validate Images ( relations retrieved from Wikidata)
Revised: Step3: Acquire and Validate Images ( relations retrieved from Wikidata)

Original: Step4:
Revised: Step4:

Original: Sample
Revised: Sample

Original: MarKG
Revised: MarKG

Original: MARS
Revised: MARS

Original: [Figure]
Revised: [Figure]

Original: ✘ Text - - ✘
Revised: KB

Original: [Figure]
Revised: [Figure]

Original: Analogy
Revised: Format RAVEN 42,000 / 14,000 / 14, ✘ Vision1,120, ✘ Classification SAT 0 / 37 /✘

Original: [Figure] ✘
Revised: Text - - ✘

Revised: Linear Word Analogy Google 0 / 50 /✘

Original: Text  - ✘
Revised: Text  - ✘

Original: Linear Word Analogy BATs 0 / 199 / [Figure] ✘
Revised: Linear Word Analogy BATs 0 / 199 / 1, ✘

Original: Text 6,  - ✘
Revised: Text 6,  - ✘

Original: Linear Word Analogy 
Revised: Linear Word Analogy E-KAR 870 / 119 /✘

Original: [Figure]
Revised: Text 2,  - ✔

Original: KAR
Revised: Multiple Choice QA MARS 10,685 / 1,228 / 1,

Original: [Figure]
Revised: [Figure]

Original: ✘ Text 2,  - ✔
Revised: Vision+Text 2,  13, ✔

Original: [Figure]
Revised: Entity Prediction which are developed from seed entities and relations in E-KAR (Chen et al., 2022a) and BATs (Gladkova et al., 2016a). Secondly, we link these seed entities into the free and open knowledge base Wikidata 3 for formalization and normalization. Thirdly, to acquire the image data, we further search from the Google engine and query from the multimodal data Laion-5B (Schuhmann et al., 2021) by the text descriptions of entities. Then, an image validation strategy is applied to filter low-quality images.

Original: 3.  D ATASET S TATISTICS
Revised: Lastly, we sample high-quality analogy data to construct MARS. A detailed description of the data collection and processing to create our datasets are in Appendix B.1 and B.2.  D ATASET S TATISTICS

Original: MARS is the evaluation dataset of the multimodal analogical reasoning task that contains analogy instances, while MarKG can provide the relative structure information of those analogy entities retrieved from Wikidata.
Revised: MARS is the evaluation dataset of the multimodal analogical reasoning task that contains analogy instances, while MarKG can provide the relative structure information of those analogy entities retrieved from Wikidata.

Original: The statistics of MARS and MarKG are shown in Table 1 and Table 5.
Revised: The statistics of MARS and MarKG are shown in Table 1 and Table 5.

Original: MarKG has 11,292 entities, 192 relations and 76,424 images, include 2,063 analogy entities and 27 analogy relations.
Revised: MarKG has 11,292 entities, 192 relations and 76,424 images, include 2,063 analogy entities and 27 analogy relations.

Original: MARS has 10,685 training, 1,228 validation and 1,415 test instances, which are more significant than previous language analogy datasets.
Revised: MARS has 10,685 training, 1,228 validation and 1,415 test instances, which are more significant than previous language analogy datasets.

Original: The original intention of MarKG is to provide prior knowledge of analogy entities and relations for better reasoning.
Revised: The original intention of MarKG is to provide prior knowledge of analogy entities and relations for better reasoning.

Original: More details including quality control can be found in Appendix B.3.
Revised: More details including quality control can be found in Appendix B.3.

Original: 3. E VALUATION M ETRICS
Revised: 3. E VALUATION M ETRICS

Original: Previous study (Chen et al., 2022a) adopts the multiple-choice QA to conduct analogical reasoning and leverage the accuracy metric for evaluation.
Revised: Previous study (Chen et al., 2022a) adopts the multiple-choice QA to conduct analogical reasoning and leverage the accuracy metric for evaluation.

Original: However, the multiple-choice QA setting may struggle to handle the one-to-more entities4 , which is very common in real-world analogy scenarios.
Revised: However, the multiple-choice QA setting may struggle to handle the one-to-more entities , which is very common in real-world analogy scenarios.

Original: Thus, we formulate the task as link prediction that directly predicts the answer entity e a ∈ E a .
Revised: Thus, we formulate the task as link prediction that directly predicts the answer entity e a ∈ E a .

Original: Our evaluation metrics include Hits@k scores (proportion of valid entities ranked in top k) and MRR (reciprocal value of the mean rank of correct entities).
Revised: Our evaluation metrics include Hits@k scores (proportion of valid entities ranked in top k) and MRR (reciprocal value of the mean rank of correct entities).

Original: More details can be found in Appendix B.4.
Revised: More details can be found in Appendix B.4.

Original:  B ENCHMARK M ETHODS
Revised:  B ENCHMARK M ETHODS

Original: In this section, we introduce some baselines to establish the initial benchmark results on MARS, including multimodal knowledge graph embedding baselines and multimodal pre-trained Transformer baselines.
Revised: In this section, we introduce some baselines to establish the initial benchmark results on MARS, including multimodal knowledge graph embedding baselines and multimodal pre-trained Transformer baselines.

Original: We further propose MarT: a multimodal analogical reasoning framework with Transformer, which can capture fine-grained associations between one analogy example and one analogy question-answer pair for better multimodal analogy abilities.
Revised: We further propose MarT: a multimodal analogical reasoning framework with Transformer, which can capture fine-grained associations between one analogy example and one analogy question-answer pair for better multimodal analogy abilities.

Original: 4. M ULTIMODAL K NOWLEDGE G RAPH E MBEDDING B ASELINES

Original: We consider three multimodal knowledge embedding (MKGE) approaches as our baselines, including IKRL (Xie et al., 2017), TransAE (Wang et al., 2019), and RSME (Wang et al., 2021).

Original: These methods are typically based on TransE (Bordes et al., 2013) or ComplEx (Trouillon et al., 2016) and combine with visual encoders to encode images for multimodal knowledge representation learning.

Original: Learnable ··· ··· ··· ··· Analogy Entity and Relation Embeddings
Revised: Learnable ··· ··· ··· ··· Analogy Entity and Relation Embeddings

Original: Analogy Example ③ Induction identical to
Revised: Analogy Example ③ Induction identical to

Original: IKRL TransAE RSME (Tesla)
Revised: IKRL TransAE RSME (Tesla)

Original: IKRL TransAE RSME (Young Tesla) ?
Revised: IKRL TransAE RSME (Young Tesla) ?

Original: ① Abduction
Revised: ① Abduction

Original: Analogy Question
Revised: Analogy Question

Original: Analogy Answer
Revised: Analogy Answer

Original: b. Pre
Revised: b. Pre

Original: -train 
Revised: -train over

Original: over MarKG c. Prompt-based Analogical Reasoning
Revised: MarKG c. Prompt-based Analogical Reasoning

Original: [PAD][PAD][CLS][MASK] [SEP] ② Mapping
Revised: [PAD][PAD][CLS][MASK] [SEP] ② Mapping

Original: Abduction
Revised: Abduction

Original: Induction
Revised: Induction

Original: S i ng l e
Revised: S i ng l e

Original: B l e nd e d [PAD] [CLS] [MASK]
Revised: B l e nd e d [PAD] [CLS] [MASK]

Original: [SEP] [CLS] [R] [SEP] [R][MASK][SEP]
Revised: [SEP] [CLS] [R] [SEP] [R][MASK][SEP]

Original: [CLS] [R] [SEP] [R][MASK][SEP] ?
Revised: [CLS] [R] [SEP] [R][MASK][SEP] ?

Original: a. MKGE Methods
Revised: a. MKGE Methods

Original: [CLS] [R] [SEP] [R][MASK][SEP] is matrix multiplication is the image of the entity is the text of the entity [PAD] is a blank image for padding
Revised: [CLS] [R] [SEP] [R][MASK][SEP] is matrix multiplication is the image of the entity is the text of the entity [PAD] is a blank image for padding

Revised: 4. M ULTIMODAL K NOWLEDGE G RAPH E MBEDDING B ASELINES

Revised: We consider three multimodal knowledge embedding (MKGE) approaches as our baselines, including IKRL (Xie et al., 2017), TransAE (Wang et al., 2019), and RSME (Wang et al., 2021).

Revised: These methods are typically based on TransE (Bordes et al., 2013) or ComplEx (Trouillon et al., 2016) and combine with visual encoders to encode images for multimodal knowledge representation learning.

Original: They can not be directly applied to the multimodal analogical reasoning task.
Revised: They can not be directly applied to the multimodal analogical reasoning task.

Original: To successfully utilize
Revised: To successfully utilize

Original: MKGE methods, we first pre-train them on MarKG to obtain entity embeddings and then follow the structure-mapping theory (Minnameier, 2010) to leverage the Abduction-Mapping-Induction as explicit pipline steps for MKGE methods.
Revised: MKGE methods, we first pre-train them on MarKG to obtain entity embeddings and then follow the structure-mapping theory (Minnameier, 2010) to leverage the Abduction-Mapping-Induction as explicit pipline steps for MKGE methods.

Original: As shown in Figure 3.a, Abduction aims to predict the relation r of ( e h , e t ) similar to the relation classification task, Mapping represents that the structural relation is mapped onto entity candidates, analogous to template-filling, and Induction utilizes the relation r to predict the tail entity of ( e q , r, ?) similar to the link prediction task.
Revised: As shown in Figure 3.a, Abduction aims to predict the relation r of ( e h , e t ) similar to the relation classification task, Mapping represents that the structural relation is mapped onto entity candidates, analogous to template-filling, and Induction utilizes the relation r to predict the tail entity of ( e q , r, ?) similar to the link prediction task.

Original: Despite the previous MKGE methods achieving excellent performance for KG-related tasks, the backbone, such as TransE, is not designed for analogy reasoning, which may hinder performance.
Revised: Despite the previous MKGE methods achieving excellent performance for KG-related tasks, the backbone, such as TransE, is not designed for analogy reasoning, which may hinder performance.

Original: Thus, we replace the backbone of MKGE methods with ANALOGY Liu et al.
Revised: Thus, we replace the backbone of MKGE methods with ANALOGY Liu et al.

Original: (2017) that models analogical structure explicitly as baselines.
Revised: (2017) that models analogical structure explicitly as baselines.

Original: 4. M ULTIMODAL P RE - TRAINED T RANSFORMER B ASELINES
Revised: 4. M ULTIMODAL P RE - TRAINED T RANSFORMER B ASELINES

Original: We select multimodal pre-trained Transformer (MPT) approaches including the single-stream models VisualBERT (Li et al., 2019), ViLT (Kim et al., 2021), the dual-stream model ViLBERT (Lu et al., 2019), and the mixed-stream model FLAVA (Singh et al., 2021) and MKGformer Chen et al.
Revised: We select multimodal pre-trained Transformer (MPT) approaches including the single-stream models VisualBERT (Li et al., 2019), ViLT (Kim et al., 2021), the dual-stream model ViLBERT (Lu et al., 2019), and the mixed-stream model FLAVA (Singh et al., 2021) and MKGformer Chen et al.

Original: (2022b)
Revised: (2022b)

Original: as the strong baselines.
Revised: as the strong baselines.

Original: However, the current multimodal pre-trained Transformer cannot directly deal with analogical reasoning.
Revised: However, the current multimodal pre-trained Transformer cannot directly deal with analogical reasoning.

Original: To address the bottleneck above, we devise an end-to-end approach to empower the MPT with analogical reasoning ability.
Revised: To address the bottleneck above, we devise an end-to-end approach to empower the MPT with analogical reasoning ability.

Original: As shown in Figure 3, we first leverage MarKG to pre-train the model over sparese MarKG to obtain the representation of entities and relations.
Revised: As shown in Figure 3, we first leverage MarKG to pre-train the model over sparese MarKG to obtain the representation of entities and relations.

Original: We then present the prompt-based analogical reasoning over MARS.
Revised: We then present the prompt-based analogical reasoning over MARS.

Original: 4.2. P RE - TRAIN OVER M AR KG
Revised: 4.2. P RE - TRAIN OVER M AR KG

Original: We represent the entities e ∈ E and relations r ∈ R as special tokens and denote E as the learnable embedding of these special tokens in the word vocabulary of language models.
Revised: We represent the entities e ∈ E and relations r ∈ R as special tokens and denote E as the learnable embedding of these special tokens in the word vocabulary of language models.

Original: In the pre-train stage, we design masked entity and relation prediction like the Masked Language Modeling (MLM) task to learn the embeddings of the special tokens over the MarKG dataset.
Revised: In the pre-train stage, we design masked entity and relation prediction like the Masked Language Modeling (MLM) task to learn the embeddings of the special tokens over the MarKG dataset.

Original: As shown in Figure 3.b, we devise a prompt template to convert the input as predicting the missing entity and relation via [MASK] token.
Revised: As shown in Figure 3.b, we devise a prompt template to convert the input as predicting the missing entity and relation via [MASK] token.

Original: In addition, we mix missing relation and entity prediction in the pre-train stage and consider different modalities of input entities.
Revised: In addition, we mix missing relation and entity prediction in the pre-train stage and consider different modalities of input entities.

Original: Specifically, we represent the visual entity e h by its image I h and special entity embedding E e h , and the text entity e t by its text description T t and special entity embedding E e t , respectively.
Revised: Specifically, we represent the visual entity e h by its image I h and special entity embedding E e h , and the text entity e t by its text description T t and special entity embedding E e t , respectively.

Original: Benefiting from the mixed entity and relation prediction with the multimodal entity in the pre-train stage, we can obtain KG embedding with multimodal semantics over the current knowledge graph MarKG.
Revised: Benefiting from the mixed entity and relation prediction with the multimodal entity in the pre-train stage, we can obtain KG embedding with multimodal semantics over the current knowledge graph MarKG.

Original: 4.2. P ROMPT - BASED A NALOGICAL R EASONING
Revised: 4.2. P ROMPT - BASED A NALOGICAL R EASONING

Original: Based on the above-pre-trained entity and relation embeddings over MarKG, we propose promptbased analogical reasoning with implicit structure mapping on downstream MARS.
Revised: Based on the above-pre-trained entity and relation embeddings over MarKG, we propose promptbased analogical reasoning with implicit structure mapping on downstream MARS.

Original: Taking the blended analogical reasoning as an example, we feed the analogy example ( I h , T t ) and analogy question-answer pair ( I q , ?) as input, and the goal is to predict the missing answer entity e a ∈ E a .
Revised: Taking the blended analogical reasoning as an example, we feed the analogy example ( I h , T t ) and analogy question-answer pair ( I q , ?) as input, and the goal is to predict the missing answer entity e a ∈ E a .

Original: We leverage an analogical prompt template to convert the input as follows:
Revised: We leverage an analogical prompt template to convert the input as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∥ represents concatenate operation in the template input,
Revised: where ∥ represents concatenate operation in the template input,

Original: I h
Revised: I h

Original: and I q represent the images of the entity e h and e q , T t is the text description of the entity e t .
Revised: and I q represent the images of the entity e h and e q , T t is the text description of the entity e t .

Original: Moreover, e h , e t , e q are entity ids and will be encoded to special entity tokens E e h , E e t , E e q in word embedding layer.
Revised: Moreover, e h , e t , e q are entity ids and will be encoded to special entity tokens E e h , E e t , E e q in word embedding layer.

Original: Since the relations are not explicitly provided in the actual analogical reasoning task, we assign [R] as a special token to denote the explicit relation between ( I h , T t ) , which is initialized with the average relation embeddings.
Revised: Since the relations are not explicitly provided in the actual analogical reasoning task, we assign [R] as a special token to denote the explicit relation between ( I h , T t ) , which is initialized with the average relation embeddings.

Original: Finally, we train the model to predict the [MASK] over the special token embedding E via cross-entropy loss, which likes the MLM task.
Revised: Finally, we train the model to predict the [MASK] over the special token embedding E via cross-entropy loss, which likes the MLM task.

Original: Remark 

Revised: Remark

Original: We summarize the two parts of T E and T A in the template as the implicit Abduction and
Revised: We summarize the two parts of T E and T A in the template as the implicit Abduction and

Original: Induction respectively, which are unified in an end-to-end learning manner with prompt tuning.
Revised: Induction respectively, which are unified in an end-to-end learning manner with prompt tuning.

Original: In addition, the analogical reasoning is reformulated as predicting the [MASK] over the multimodal analogy entity embeddings to obtain e a .
Revised: In addition, the analogical reasoning is reformulated as predicting the [MASK] over the multimodal analogy entity embeddings to obtain e a .

Original: 4. M AR T: A M ULTIMODAL A NALOGICAL R EASONING F RAMEWORK WITH T RANSFORMER
Revised: 4. M AR T: A M ULTIMODAL A NALOGICAL R EASONING F RAMEWORK WITH T RANSFORMER

Original: [Figure]
Revised: [Figure]

Revised: [R] [R] [R] [R]

Original: Although the approach above-mentioned can enable multimodal pre-trained Transformer models to multimodal analogical reasoning, they only superficially consider implicit Abduction and Induction , ignoring the fine-grained associations between the analogy example and analogy question-answer pair.
Revised: Although the approach above-mentioned can enable multimodal pre-trained Transformer models to multimodal analogical reasoning, they only superficially consider implicit Abduction and Induction , ignoring the fine-grained associations between the analogy example and analogy question-answer pair.

Original: Adaptive Interaction Across Analogy.
Revised: Adaptive Interaction Across Analogy.

Original: Since the analogy question may interfere with the represen- tation of the analogy example and the inevitable noisy data issue, we propose adaptive interaction across analogy in encoding process to interact between the analogy example and question-answer pair adaptively, as shown in Figure 4.
Revised: Since the analogy question may interfere with the represen- tation of the analogy example and the inevitable noisy data issue, we propose adaptive interaction across analogy in encoding process to interact between the analogy example and question-answer pair adaptively, as shown in Figure 4.

Original: Denote the input to a Transformer layer as X =
Revised: Denote the input to a Transformer layer as X =

Original: [ X E ∥ X A ] , where X E and X
Revised: [ X E ∥ X A ] , where X E and X

Original: A denote the hidden representation of analogy example T E and question-answer pair T A respectively.
Revised: A denote the hidden representation of analogy example T E and question-answer pair T A respectively.

Original: In each attention head of layer, the query and key representation can be formalized as:
Revised: In each attention head of layer, the query and key representation can be formalized as:

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K are project matrices.
Revised: where W Q , W K are project matrices.

Original: A similar expression also holds for values V .
Revised: A similar expression also holds for values V .

Original: Then the attention probability matrix P can be defined in terms of four sub-matrices:
Revised: Then the attention probability matrix P can be defined in terms of four sub-matrices:

Original: [Equation]
Revised: [Equation]

Original: where P EE , P AA (diagonal of P ) are intra-analogy attentions and P EA , P AE (anti-diagonal of P ) are inter-analogy attentions.
Revised: where P EE , P AA (diagonal of P ) are intra-analogy attentions and P EA , P AE (anti-diagonal of P ) are inter-analogy attentions.

Original: We use the gate G to regulate the inter-analogy interactions adaptively:
Revised: We use the gate G to regulate the inter-analogy interactions adaptively:

Original: [Equation]
Revised: [Equation]

Original: ) where G ∈ R 2 × 2 is adaptive association gate which has two learnable variables g EA , g AE ∈ [0 , 1] . 
Revised: where G ∈ R 2 × 2 is adaptive association gate which has two learnable variables g EA , g AE ∈ [0 , 1] . [Table]

Revised: to-end baselines with multimodal pretrained Transformer (MPT) methods and obtain the best performance in MarT MKGformer.

Original: Remark 

Revised: Remark

Original: On the one hand, the query from T A may interfere with the example from T E .
Revised: On the one hand, the query from T A may interfere with the example from T E .

Original: On the other hand, T E may have a weaker impact on T A in noisy data.
Revised: On the other hand, T E may have a weaker impact on T A in noisy data.

Original: Adaptive association gates can increase and decrease inter-analogy interaction automatically based on the intimacy of T E and T A .
Revised: Adaptive association gates can increase and decrease inter-analogy interaction automatically based on the intimacy of T E and T A .

Original: Relation-Oriented Structure Mapping.
Revised: Relation-Oriented Structure Mapping.

Original: The structure mapping theory emphasizes the relation transfer rather than object similarity in analogical reasoning, it is relations between objects, rather[Table] We also utilize our MarT on end-to-end baselines with multimodal pretrained Transformer (MPT) methods and obtain the best performance in MarT MKGformer. than attributes of objects, are mopped from base to target .
Revised: The structure mapping theory emphasizes the relation transfer rather than object similarity in analogical reasoning, it is relations between objects, rather than attributes of objects, are mopped from base to target .

Original: For example, battery can make an analogy to reservoir because they both store potential, rather than their shapes being cylindrical.
Revised: For example, battery can make an analogy to reservoir because they both store potential, rather than their shapes being cylindrical.

Original: Motivated by this, we propose the relaxation loss to bring the relations closer and alienate the entities:
Revised: Motivated by this, we propose the relaxation loss to bring the relations closer and alienate the entities:

Original: [Equation]
Revised: [Equation]

Original: where |S| is the total number of the training set S , h E [R] is the hidden feature of [R] in analogy example T E output from the MLM head, sim ( · ) is the cosine similarity.
Revised: where |S| is the total number of the training set S , h E [R] is the hidden feature of [R] in analogy example T E output from the MLM head, sim ( · ) is the cosine similarity.

Original: We leverage the masked entity prediction task to obtain the answer entity e a with a cross-entropy loss:
Revised: We leverage the masked entity prediction task to obtain the answer entity e a with a cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Original: Afterwards, we interpolate the relaxation loss L rel and the masked entity prediction loss L mem using parameter λ to produce the final loss L :
Revised: Afterwards, we interpolate the relaxation loss L rel and the masked entity prediction loss L mem using parameter λ to produce the final loss L :

Original: [Equation]
Revised: [Equation]

Original: Remark 

Revised: Remark

Original: The relaxation loss is composed of pull-in and pull-away that correspond to the close relation and alienate entity terms, respectively, which can constrain the model’s focus on relation structure transfer and implicitly realize the Structure Mapping process.
Revised: The relaxation loss is composed of pull-in and pull-away that correspond to the close relation and alienate entity terms, respectively, which can constrain the model’s focus on relation structure transfer and implicitly realize the Structure Mapping process.

Original:  R ESULTS AND A NALYSIS 5. M AIN R ESULTS
Revised:  R ESULTS AND A NALYSIS 5. M AIN R ESULTS

Original: The main performance results of all benchmark methods can be seen in Table 2.
Revised: The main performance results of all benchmark methods can be seen in Table 2.

Original: In general, we find the performance of multimodal knowledge graph embedding (MKGE) baselines and multimodal pre-trained Transformer (MPT) baselines is comparable except MKGformer, which establishes a  competitive baseline of MARS.
Revised: In general, we find the performance of multimodal knowledge graph embedding (MKGE) baselines and multimodal pre-trained Transformer (MPT) baselines is comparable except MKGformer, which establishes a [Table] competitive baseline of MARS.

Original: In addition, when replacing the backbone of MKGE methods with
Revised: In addition, when replacing the backbone of MKGE methods with

Original: ANALOGY that models analogical structure explicitly, the performance is significantly improved.
Revised: ANALOGY that models analogical structure explicitly, the performance is significantly improved.

Original: Meanwhile, the MPT models without analogy-related structures obtain substantial performance with the analogical reasoning ability enhanced by MarT. For example, although MKGformer achieves[Table] outstanding performance, MarT MKGformer further improves and obtains state-of-the-art performance, exceeding other methods by 4.9%-12.4% points in the MRR metric.
Revised: Meanwhile, the MPT models without analogy-related structures obtain substantial performance with the analogical reasoning ability enhanced by MarT. For example, although MKGformer achieves outstanding performance, MarT MKGformer further improves and obtains state-of-the-art performance, exceeding other methods by 4.9%-12.4% points in the MRR metric.

Original: It reveals that the MarT framework stimulates the ability of the Transformer-based model for multimodal analogical reasoning.
Revised: It reveals that the MarT framework stimulates the ability of the Transformer-based model for multimodal analogical reasoning.

Original: We also report the pre-training results on MarKG in Appendix C.2.
Revised: We also report the pre-training results on MarKG in Appendix C.2.

Original: 5. G ENERALIZE TO N OVEL R ELATION
Revised: 5. G ENERALIZE TO N OVEL R ELATION

Original: [Table]
Revised: [Table]

Original: Making analogies from one domain to another novel domain is a fundamental ingredient for human creativity.
Revised: Making analogies from one domain to another novel domain is a fundamental ingredient for human creativity.

Original: In this section, we conduct a novel relation transfer experiment (including both task settings) to measure how well the models generalize by analogy to unfamiliar relations.
Revised: In this section, we conduct a novel relation transfer experiment (including both task settings) to measure how well the models generalize by analogy to unfamiliar relations.

Original: Specifically, we randomly split the 27 analogy relations into the source and target relations.
Revised: Specifically, we randomly split the 27 analogy relations into the source and target relations.

Original: The models are then trained on the source and tested on the novel target relations.
Revised: The models are then trained on the source and tested on the novel target relations.

Original: As shown in Table 3, we observe that
Revised: As shown in Table 3, we observe that

Original: MarT MKGformer can indeed learn to make sense of unfamiliar relations, respectively.
Revised: MarT MKGformer can indeed learn to make sense of unfamiliar relations, respectively.

Original: We further evaluate the model without pre-training on MarKG and find the performance decreased, which indicates that the structure knowledge provided by MarKG is critical for generalization.
Revised: We further evaluate the model without pre-training on MarKG and find the performance decreased, which indicates that the structure knowledge provided by MarKG is critical for generalization.

Original: Note that the novel relation transfer setting is somewhat similar to zero-shot or domain generalization, and we hope our work can benefit other communities.
Revised: Note that the novel relation transfer setting is somewhat similar to zero-shot or domain generalization, and we hope our work can benefit other communities.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: To validate the effectiveness of MarKG and MarT, we conduct an ablation study as shown in Table 4.
Revised: To validate the effectiveness of MarKG and MarT, we conduct an ablation study as shown in Table 4.

Original: We observe that discarding pre-train on MarKG results in worse performance for both MKGE and MPT baselines.
Revised: We observe that discarding pre-train on MarKG results in worse performance for both MKGE and MPT baselines.

Original: It indicates that the knowledge structure information provided by MarKG helps learn the representation of entities and relations, further benefiting analogical reasoning.
Revised: It indicates that the knowledge structure information provided by MarKG helps learn the representation of entities and relations, further benefiting analogical reasoning.

Original: We also find that the performance clearly drops when ablating each component of MarT and reaches the valley when ablating all, proving the effectiveness of each analogical component of our MarT.
Revised: We also find that the performance clearly drops when ablating each component of MarT and reaches the valley when ablating all, proving the effectiveness of each analogical component of our MarT.

Original: Moreover, we ablate the analogy example in the input and find the performance drops a lot, which reveals the importance of analogical prompts.
Revised: Moreover, we ablate the analogy example in the input and find the performance drops a lot, which reveals the importance of analogical prompts.

Original: 5. A NALYSIS
Revised: 5. A NALYSIS

Original: Analysis across Different Sub-Tasks.
Revised: Analysis across Different Sub-Tasks.

Original: In previous Table 2, we are amazed by ANALOGY significantly improving the performance of MKGE baselines.
Revised: In previous Table 2, we are amazed by ANALOGY significantly improving the performance of MKGE baselines.

Original: Therefore, we further compare the performance of vanilla baselines to the addition of analogical components in different sub-task settings. As shown in Figure 5, we observe that vanilla TransAE performs poorly in the blended task setting. However, when replacing the backbone TransE with ANALOGY, TransAE is competent in blended analogical reasoning setting and even outperforms the single setting. On the other side, RSME with
Revised: Therefore, we further compare the perfor-

Original: Panax notoginseng ?
Revised: Panax notoginseng ?

Original: Traditional Chinese Medicine instance of
Revised: Traditional Chinese Medicine instance of

Original: Apple fruit instance of battle ?
Revised: Apple fruit instance of battle ?

Original: campaign Inland Lake Qinghai Lake correspond to
Revised: campaign Inland Lake Qinghai Lake correspond to

Original: MKGformer
Revised: MKGformer

Original: MKGformer*
Revised: MKGformer*

Original: TransAE
Revised: TransAE

Original: TransAE* film increment scheme battle war siege life aircraft ocean battle reaction court correspond to
Revised: TransAE* film increment scheme battle war siege life aircraft ocean battle reaction court correspond to

Original: Top-3 Entity
Revised: Top-3 Entity

Original: Gold Rank
Revised: Gold Rank

Original: MKGformer
Revised: MKGformer

Original: MKGformer*
Revised: MKGformer*

Original: TransAE
Revised: TransAE

Original: TransAE* bread capital phone fruit driedfruit citrus
Revised: TransAE* bread capital phone fruit driedfruit citrus

Original: Citrus grain shipping plant Citrus dessert
Revised: Citrus grain shipping plant Citrus dessert

Original: Analogical Example
Revised: Analogical Example

Original: Question-Answer Pair
Revised: Question-Answer Pair

Original: [Figure]
Revised: [Figure]

Revised: mance of vanilla baselines to the addition of analogical components in different sub-task settings.

Revised: As shown in Figure 5, we observe that vanilla TransAE performs poorly in the blended task setting.

Revised: However, when replacing the backbone TransE with ANALOGY, TransAE is competent in blended analogical reasoning setting and even outperforms the single setting.

Revised: On the other side, RSME with

Original: ComplEx as backbone can handle the blended setting reluctantly but perform worse than the single setting.
Revised: ComplEx as backbone can handle the blended setting reluctantly but perform worse than the single setting.

Original: ANALOGY improves the performance of RSME in this situation.
Revised: ANALOGY improves the performance of RSME in this situation.

Original: Meanwhile, MarT further explores the potential of MKGformer and improves its performance in various tasks.
Revised: Meanwhile, MarT further explores the potential of MKGformer and improves its performance in various tasks.

Original: All in all, the analogical components consistently improve the multimodal analogical reasoning ability of all baseline methods, especially in blended analogical reasoning, which supports Mayer’s theory (Mayer, 2002) that analogical reasoning is more affinity for multimodal scenarios . 
Revised: All in all, the analogical components consistently improve the multimodal analogical reasoning ability of all baseline methods, especially in blended analogical reasoning, which supports Mayer’s theory (Mayer, 2002) that analogical reasoning is more affinity for multimodal scenarios . [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: q

Revised: [Equation]

Revised: [Figure]

Original: Case Analysis.
Revised: Case Analysis.

Original: As shown in Figure 6, we provide case analysis and observe that the top ranking entities (film, life, etc.) of the baselines without analogical components are usually irrelevant to the question entity “campaign”.
Revised: As shown in Figure 6, we provide case analysis and observe that the top ranking entities (film, life, etc.) of the baselines without analogical components are usually irrelevant to the question entity “campaign”.

Original: Analogical components make the predictions more reasonable and successfully predict the answer entity “battle”.
Revised: Analogical components make the predictions more reasonable and successfully predict the answer entity “battle”.

Original: In the difficult blended analogical reasoning setting, the blended modal input of visual and text is challenging.
Revised: In the difficult blended analogical reasoning setting, the blended modal input of visual and text is challenging.

Original: We find that vanilla MKGformer and TransAE fail to understand the visual semantic of “apple” and incorrectly linked with “capital, phone, shipping” that related to “Apple Company”.
Revised: We find that vanilla MKGformer and TransAE fail to understand the visual semantic of “apple” and incorrectly linked with “capital, phone, shipping” that related to “Apple Company”.

Original: We also notice that TransAE with ANALOGY as backbone significantly decreases the prediction error but incorrectly predicts “plant” as the top-1 entity due to the interference of “Panax notoginseng”.
Revised: We also notice that TransAE with ANALOGY as backbone significantly decreases the prediction error but incorrectly predicts “plant” as the topentity due to the interference of “Panax notoginseng”.

Original: On the contrary, MarT MKGformer with relaxation loss can alienate the entities and focus on relation structures transfer and obtain reasonable predictions.
Revised: On the contrary, MarT MKGformer with relaxation loss can alienate the entities and focus on relation structures transfer and obtain reasonable predictions.

Original: These observations reveal that multimodal analogical reasoning is a highly challenging task, and analogyaware components could enhance the analogical ability of models.
Revised: These observations reveal that multimodal analogical reasoning is a highly challenging task, and analogy-aware components could enhance the analogical ability of models.

Original: Besides, we discuss limitations in Appendix A and provide a comprehensive error analysis in Appendix D.
Revised: Besides, we discuss limitations in Appendix A and provide a comprehensive error analysis in Appendix D.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this work, we introduce the new task of multimodal analogical reasoning over knowledge graphs with a new dataset and multimodal knowledge graph.
Revised: In this work, we introduce the new task of multimodal analogical reasoning over knowledge graphs with a new dataset and multimodal knowledge graph.

Original: Preliminary experiments show that this task brings a rather difficult challenge and is worth further exploration.
Revised: Preliminary experiments show that this task brings a rather difficult challenge and is worth further exploration.

Original: Besides evaluating the analogical reasoning ability of models, there are some potential applications to explore: (1) knowledge graph completion with analogies, (2) transfer learning and zero-shot learning by analogy and (3) analogical question answering.
Revised: Besides evaluating the analogical reasoning ability of models, there are some potential applications to explore: (1) knowledge graph completion with analogies, (2) transfer learning and zero-shot learning by analogy and (3) analogical question answering.

Original: We hope our work inspires future research on analogical reasoning and applications, especially in the multimodal world.
Revised: We hope our work inspires future research on analogical reasoning and applications, especially in the multimodal world.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The source MARS and MarKG datasetsare in the supplementary materials and willcertainly be released.
Revised: The source MARS and MarKG datasets will be released on Github soon.

Original: In order to provide support to reproduce our experiments in Section 5, we provide the detailed source code of all pipeline baselines (IKRL, TransAE, RSME) and end-to-end baselines (VisualBERT, ViLBERT, ViLT, FLAVA, MKGformer) in the supplementary materials with all scripts and hyper-parameters.
Revised: In order to provide support to reproduce our experiments in Section 5, we provide the detailed source code of all pipeline baselines (IKRL, TransAE, RSME) and end-to-end baselines (VisualBERT, ViLBERT, ViLT, FLAVA, MKGformer) in the supplementary materials with all scripts and hyper-parameters.

Original: We also provide a README script to instruct how to run the codes
Revised: We also provide a README script to instruct how to run the codes.

Original: A L IMITATIONS
Revised: A CKNOWLEDGMENT

Revised: We would like to express gratitude to the anonymous reviewers for their kind comments.

Revised: This work was supported by the National Natural Science Foundation of China (No.62206246 and U19B2027),

Revised: Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural

Revised: Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G), and CAAI-Huawei MindSpore Open Fund.

Revised: -acl.

Revised: URL what alexnet is to CV: can pre-trained language models identify analogies?

Revised: In Chengqing Zong

Revised: _Training_Multi

Revised: Modal

Revised: _CVPR_2020_

Revised: A L IMITATIONS

Original: The proposed work still has some limitations.
Revised: The proposed work still has some limitations.

Original: We try to simulate the real-world multimodal analogy reasoning setting; however, it still can not predict analogical entities that are not existing in the multimodal knowledge graph.
Revised: We try to simulate the real-world multimodal analogy reasoning setting; however, it still can not predict analogical entities that are not existing in the multimodal knowledge graph.

Original: Such an issue is also known as inductive knowledge graph completion, and we leave this for future works.
Revised: Such an issue is also known as inductive knowledge graph completion, and we leave this for future works.

Original: Besides we have not evaluated the very large-scale pre-trained models on the MARS due to the GPU resources and it is well worth investigating whether largescale pre-trained models can emerge the multimodal analogy reasoning ability
Revised: Besides, we have not evaluated the very large-scale pre-trained models on the MARS due to the GPU resources, and it is well worth investigating whether largescale pre-trained models can emerge the multimodal analogy reasoning ability.

Original: B A DDITIONAL D ATASETS I NFORMATION B. D ATASET C ONSTRUCTION
Revised: B A DDITIONAL D ATASETS I NFORMATION B. D ATASET C ONSTRUCTION

Original: Step 1: Collect Analogy Entities and Relations.
Revised: Step 1: Collect Analogy Entities and Relations.

Original: Since E-KAR and BATs are widely used text analogy datasets with high-quality and semantically specific entities, we collect the analogy seed  entities E a and relations from them according to the following criteria: (1) The relations and entities that have the same meanings will be merged.
Revised: Since E-KAR and BATs are widely used text analogy datasets with high-quality and semantically specific entities, we collect the analogy seed has cause part of corresponds to instance of juxtaposi1on to loca1on opposite of tool of has quality follow made from material synonym intersec1on to head-modiﬁer has use antonym verb-object iden1cal to subject-object diﬀerent from prerequisite contradictory to target of subject-predicate takes place in probabilis1c a>ribute metaphor entities E a and relations from them according to the following criteria: (1) The relations and entities that have the same meanings will be merged.

Original: For example, we merge the relation is a of E-KAR and the relation Hypernyms of BATs since they both represent the hypernym relationship of entities.
Revised: For example, we merge the relation is a of E-KAR and the relation Hypernyms of BATs since they both represent the hypernym relationship of entities.

Original: We obtain 38 relations after this step.
Revised: We obtain 38 relations after this step.

Original: The relation must imply analogical knowledge reasoning rather than simple word linear analogy.
Revised: The relation must imply analogical knowledge reasoning rather than simple word linear analogy.

Original: For example, we discard the analogy relations that only reflect simple word changes of BATs dataset such as Inflections (Nouns, Verbs, etc.) and Derivation (Stem change, etc.) .
Revised: For example, we discard the analogy relations that only reflect simple word changes of BATs dataset such as Inflections (Nouns, Verbs, etc.) and Derivation (Stem change, etc.) .

Original: After this step, we filter 11 relations and retain 27 analogy relations.
Revised: After this step, we filter 11 relations and retain 27 analogy relations.

Original: has cause part of corresponds to instance of juxtaposi1on to loca1on opposite of tool of has quality follow made from material synonym intersec1on to head-modiﬁer has use antonym verb-object iden1cal to subject-object diﬀerent from prerequisite contradictory to target of subject-predicate takes place in probabilis1c a>ribute metaphor

Original: (3) The entity must be visualizable and realistic.
Revised: (3) The entity must be visualizable and realistic.

Original: We filter those entities that cannot be linked into
Revised: We filter those entities that cannot be linked into

Original: Wikidata and drop out the extremely abstract entities such as virtue by hand (some entities that have no image after Step 3 are also filtered).
Revised: Wikidata and drop out the extremely abstract entities such as virtue by hand (some entities that have no image after Step 3 are also filtered).

Original: We discard a total of 463 entities after filtering.
Revised: We discard a total of 463 entities after filtering.

Original: Finally, we obtained 2,063 seed entities and 27 relations.
Revised: Finally, we obtained 2,063 seed entities and 27 relations.

Original: Step 2: Link to Wikidata and Retrieve Neighbors.
Revised: Step 2: Link to Wikidata and Retrieve Neighbors.

Original: Consider that complex analogical reasoning is difficult through individual information (descriptions or images) of entities.
Revised: Consider that complex analogical reasoning is difficult through individual information (descriptions or images) of entities.

Original: We link the analogy seed entities to Wikidata by Mediawiki API 5 and retrieve the one-hop neighbors of seed entities as well as the possible relationships between the seed entities to obtain their neighbor structure information.
Revised: We link the analogy seed entities to Wikidata by Mediawiki API 4 and retrieve the one-hop neighbors of seed entities as well as the possible relationships between the seed entities to obtain their neighbor structure information.

Original: In this step, we also take the retrieved descriptions from Wikidata as the textual information of entities and relations.
Revised: In this step, we also take the retrieved descriptions from Wikidata as the textual information of entities and relations.

Original: Step 3: Acquire and Validate Images.
Revised: Step 3: Acquire and Validate Images.

Original: We collect images from two sources: Google Engine and
Revised: We collect images from two sources: Google Engine and

Original: Laion-5B query service 6 .
Revised: Laion-5B query service 5 .

Original: We search from Google Engine with the descriptions of entities and crawl 5 images per entity.
Revised: We search from Google Engine with the descriptions of entities and crawl 5 images per entity.

Original: Laion-5B service depends on Clip retrieval and query by k nn index; we leverage the clip text embedding of the description and also query 5 images for each entity.
Revised: Laion-5B service depends on Clip retrieval and query by k nn index; we leverage the clip text embedding of the description and also query 5 images for each entity.

Original: Then we apply four filters to the above images: (1) we check the format of the images and filter invalid files, (2) we remove corrupted (the images are damaged and cannot be opened), low-quality (image size less than 50 × 50 or non-panchromatic images) and duplicate images, (3) we use CLIP (Radford et al., 2021) to remove the images with outlier visual embeddings, (4) we delete unreasonable images manually.
Revised: Then we apply four filters to the above images: (1) we check the format of the images and filter invalid files, (2) we remove corrupted (the images are damaged and cannot be opened), low-quality (image size less than 50 × 50 or non-panchromatic images) and duplicate images, (3) we use CLIP (Radford et al., 2021) to remove the images with outlier visual embeddings, (4) we delete unreasonable images manually.

Original: Step 4: Sample Analogical Reasoning Data.
Revised: Step 4: Sample Analogical Reasoning Data.

Original: From Step 1 to Step 3, we obtain the MarKG, which includes 2,063 analogy entities, 8,881 neighbor entities, 27 analogy relations and 165 other relations.
Revised: From Step 1 to Step 3, we obtain the MarKG, which includes 2,063 analogy entities, 8,881 neighbor entities, 27 analogy relations and 165 other relations.

Original: To construct the MARS dataset, we sample analogy example ( e h , r, e t ) and analogy question-answer pair ( e q , r, e a ) with the same relation r from 2,063 analogy entities, but we do not explicitly provide the relation in the input.
Revised: To construct the MARS dataset, we sample analogy example ( e h , r, e t ) and analogy question-answer pair ( e q , r, e a ) with the same relation r from 2,063 analogy entities, but we do not explicitly provide the relation in the input.

Original: Then we split the data into different task settings evenly.
Revised: Then we split the data into different task settings evenly.

Original: More details about the sample strategy of MARS can be seen in Section B.2.
Revised: More details about the sample strategy of MARS can be seen in Section B.2.

Original: B. S AMPLE S TRATEGY OF MARS
Revised: B. S AMPLE S TRATEGY OF MARS

Original: In Section B.1, we obtain the analogy seed entities E a and the analogy relations between E a .
Revised: In Section B.1, we obtain the analogy seed entities E a and the analogy relations between E a .

Original: Then we sample analogy example ( e h , e t ) and analogy question-answer pair ( e q , e a ) from E a .
Revised: Then we sample analogy example ( e h , e t ) and analogy question-answer pair ( e q , e a ) from E a .

Original: Guided by SMT, we make sure that ( e h , e t ) and ( e q , e a ) have the same relation r .
Revised: Guided by SMT, we make sure that ( e h , e t ) and ( e q , e a ) have the same relation r .

Original: Specifically, we divide the entity pairs that share the same relation into two categories to avoid overlap issues.
Revised: Specifically, we divide the entity pairs that share the same relation into two categories to avoid overlap issues.

Original: Then we randomly sample the analogy examples from one category and the analogy question-answer pairs from another to construct analogy input instances.
Revised: Then we randomly sample the analogy examples from one category and the analogy question-answer pairs from another to construct analogy input instances.

Original: Last, we split the instances into different task settings evenly.
Revised: Last, we split the instances into different task settings evenly.

Original: B. D ATASET D ETAILS
Revised: B. D ATASET D ETAILS

Original: The statistical comparison of MarKG with two multimodal knowledge graph datasets WN9IMG (Xie et al., 2017) and FB15k-IMG (Liu et al., 2019) as shown in Table 5, we report the number

Original: [Table]
Revised: [Table]

Original:  of entity, relation, triple, image and the data source.
Revised: The statistical comparison of MarKG with two multimodal knowledge graph datasets WN9IMG (Xie et al., 2017) and FB15k-IMG (Liu et al., 2019) as shown in Table 5, we report the number of entity, relation, triple, image and the data source.

Original: Note that WN9-IMG and FB15k-IMG aim for knowledge completion and triple classification tasks while our MarKG aims to support MARS to do multimodal analogical reasoning.
Revised: Note that WN9-IMG and FB15k-IMG aim for knowledge completion and triple classification tasks while our MarKG aims to support MARS to do multimodal analogical reasoning.

Original: We also show the complete relations of our MARS in Table 6 and the distribution of relation categories in Figure7. [Table] Example part of
Revised: We also show the complete relations of our MARS in Table 6 and the distribution of relation categories in Figure

Original: Object of which the subject is a part.

Original: mouse : computer corresponds to

Original: Terms generally correspond to each other.

Original: entrepreneur : laborer juxtaposition to

Original: Two terms belong to the same hypernym or have the same properties or functions.

Original: child : minor synonym

Original: Sense of another lexeme with the same meaning as this sense.

Original: tired : exhausted made from material

Original: Material the subject or the object is made of or derived from.

Original: building

Original: [Table]

Original: cement antonym

Original: Sense of a lexeme with the opposite meaning to this

Original: [Table]

Original: [Table]

Original: cool

Original: [Table]

Original: Underlying

Original: [Table]

Original: thing

Original: [Table]

Original: ultimately

Original: [Table]

Original: this effect.

Original: cleaning :

Original: [Table]

Original: opposite of

Original: Item

Original: [Table]

Original: [Table]

Original: The meanings

Original: [Table]

Original: terms

Original: [Table]

Original: identical.

Original: [Table]

Original: The preceding term modifies the other.

Original: [Table]
Revised: [Table]

Original: different

Original: [Table]

Original: Quality Control of Datasets.
Revised: Quality Control of Datasets.

Original: We devise some quality control strategies while construct our MarKG and MARS datasets: (1) Entity and relation formalization and normalization.
Revised: We devise some quality control strategies while construct our MarKG and MARS datasets: (1) Entity and relation formalization and normalization.

Original: We link the analogy entities collected from E-KAR and SAT to Wikidata and filter non-link items.
Revised: We link the analogy entities collected from E-KAR and SAT to Wikidata and filter non-link items.

Original: Since Wikidata is a knowledge base with quality-assured, some rare or worthless entities are excluded.
Revised: Since Wikidata is a knowledge base with quality-assured, some rare or worthless entities are excluded.

Original: (2) Image validation mechanism.
Revised: (2) Image validation mechanism.

Original: We devise complex image filter strategies to control the robustness of image data, as mentioned in Section B.1.
Revised: We devise complex image filter strategies to control the robustness of image data, as mentioned in Section B.1.

Original: (3) Control of text description.
Revised: (3) Control of text description.

Original: We take the description in
Revised: We take the description in

Original: Wikidata as the textual information of entities.
Revised: Wikidata as the textual information of entities.

Original: [Table]

Original: Human evaluation on MARS.
Revised: Human evaluation on MARS.

Original: To evaluate the complexity and difficulty of the multimodal analogical reasoning task, we build a human evaluation in this section.
Revised: To evaluate the complexity and difficulty of the multimodal analogical reasoning task, we build a human evaluation in this section.

Original: However, humans encounter the  following problems in this entity prediction task: (1) The candidate entity set is too huge for humans to select one entity.
Revised: However, humans encounter the [Table] following problems in this entity prediction task: (1) The candidate entity set is too huge for humans to select one entity.

Original: (2) Hit@k metric is not available since human hard rank predictions.
Revised: (2) Hit@k metric is not available since human hard rank predictions.

Original: Therefore, we utilize the multiple-choice format for human beings and apply the Accuracy metric to evaluate.
Revised: Therefore, we utilize the multiple-choice format for human beings and apply the Accuracy metric to evaluate.

Original: Specifically, we randomly sample 100 instances from the test set to construct the evaluation set, and we use the top 10 ranking entities in TransAE prediction as candidate choices for each instance.
Revised: Specifically, we randomly sample 100 instances from the test set to construct the evaluation set, and we use the top 10 ranking entities in TransAE prediction as candidate choices for each instance.

Original: If the golden answer entity is not in the top 10 entities, we will randomly replace one candidate with the golden entity.
Revised: If the golden answer entity is not in the top 10 entities, we will randomly replace one candidate with the golden entity.

Original: Then humans must select one entity from the candidate choices as the answer entity.
Revised: Then humans must select one entity from the candidate choices as the answer entity.

Original: The results can be seen in Table 7.
Revised: The results can be seen in Table 7.

Original: We limit the prediction space of baseline models in candidate choices for a fair comparison.
Revised: We limit the prediction space of baseline models in candidate choices for a fair comparison.

Original: We find that the performance of the baselines in the Hit@1 metric has a large gap with human, which indicates the difficulty of the multimodal analogical reasoning task.
Revised: We find that the performance of the baselines in the Hit@1 metric has a large gap with human, which indicates the difficulty of the multimodal analogical reasoning task.

Original: B. D ETAILED E VALUATION M ETRICS
Revised: B. D ETAILED E VALUATION M ETRICS

Original: The evaluation method of (Chen et al., 2022a) can not reflect one-to-more entities and does not fully explore the internal knowledge in the models due to the limited search space.
Revised: The evaluation method of (Chen et al., 2022a) can not reflect one-to-more entities and does not fully explore the internal knowledge in the models due to the limited search space.

Original: Thus, we follow the link prediction task and choose Hits@k and MRR as our evaluation metrics.
Revised: Thus, we follow the link prediction task and choose Hits@k and MRR as our evaluation metrics.

Original: Both metrics are in the range [0 , 1] .
Revised: Both metrics are in the range [0 , 1] .

Original: The bigger, the better performance.
Revised: The bigger, the better performance.

Original: The Hits at k metric (Hits@k) is acquired by counting the number of times the golden entity appears at the first k positions in the predictions.
Revised: The Hits at k metric (Hits@k) is acquired by counting the number of times the golden entity appears at the first k positions in the predictions.

Original: Given the prediction score of each entity in the candidate entity set, we sort the score and obtain the ranking of each entity.
Revised: Given the prediction score of each entity in the candidate entity set, we sort the score and obtain the ranking of each entity.

Original: Denote the rank of the gold entity of i triple as rank i , and the reciprocal rank is 1 /rank i .
Revised: Denote the rank of the gold entity of i triple as rank i , and the reciprocal rank is 1 /rank i .

Original: The Mean Reciprocal Rank (MRR) is the average of the reciprocal ranks across all triples in the knowledge graph:
Revised: The Mean Reciprocal Rank (MRR) is the average of the reciprocal ranks across all triples in the knowledge graph:

Original: [Equation]
Revised: [Equation]

Original: where |S| is the total number of the training set
Revised: where |S| is the total number of the training set.

Original: [Table]

Original: C A DDITIONAL OF E XPERIMENTS C. I MPLEMENTATION D ETAILS
Revised: C A DDITIONAL OF E XPERIMENTS C. I MPLEMENTATION D ETAILS

Original: [Table]
Revised: [Table]

Original: This section detail the training procedures and hyper-parameters for various models.
Revised: This section detail the training procedures and hyper-parameters for various models.

Original: For multimodal knowledge representation methods, we first use MarKG to do knowledge representation learning and obtain the entity and relation matrix embeddings.
Revised: For multimodal knowledge representation methods, we first use MarKG to do knowledge representation learning and obtain the entity and relation matrix embeddings.

Original: Then we apply abduction and induction processes to continue training the models on the MARS dataset.
Revised: Then we apply abduction and induction processes to continue training the models on the MARS dataset.

Original: Note that these processes are serial and share models.
Revised: Note that these processes are serial and share models.

Original: For multimodal pre-trained Transformer models, we also use MarKG to pre-train the models and then fine-tune on MARS end-to-end with our analogy prompt tuning strategy.
Revised: For multimodal pre-trained Transformer models, we also use MarKG to pre-train the models and then fine-tune on MARS end-to-end with our analogy prompt tuning strategy.

Original: We utilize
Revised: We utilize

Original: Pytorch to conduct all experiments with 1 Nvidia 3090 GPU.
Revised: Pytorch to conduct all experiments with 1 Nvidia 3090 GPU.

Original: The details of hyper-parameters can be seen in Table 
Revised: The details of hyper-parameters can be seen in Table 8.

Revised: C. R ESULTS OF P RE - TRAINING ON M AR KG.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: C. R ESULTS OF P RE - TRAINING ON M AR KG.

Original: We report the pre-train results on MarKG in Table 9.
Revised: We report the pre-train results on MarKG in Table 9.

Original: We find that MPT baselines perform better than MKGE baselines consistently.
Revised: We find that MPT baselines perform better than MKGE baselines consistently.

Original: It reveals the strong fit ability of Transformer-based models.
Revised: It reveals the strong fit ability of Transformer-based models.

Original: As shown in Figure 8, we can observe that pre-training and fine-tuning stages trends are roughly the same, especially in the same type of baselines, which indicates that pre-train on MarKG benefits analogical reasoning on MARS.
Revised: As shown in Figure 8, we can observe that pre-training and fine-tuning stages trends are roughly the same, especially in the same type of baselines, which indicates that pre-train on MarKG benefits analogical reasoning on MARS.

Original: C. R ESULTS OF I MPLICIT R ELATION I NFERENCE OF MPT.
Revised: C. R ESULTS OF I MPLICIT R ELATION I NFERENCE OF MPT.

Original: [Table]
Revised: [Table]

Original: We conduct an evaluation experiment on the relation inference of MKGE and MPT methods.
Revised: We conduct an evaluation experiment on the relation inference of MKGE and MPT methods.

Original: For
Revised: For

Original: MKGE methods, we evaluate the relation predicted of Abuduction process with hit@k metrics.
Revised: MKGE methods, we evaluate the relation predicted of Abuduction process with hit@k metrics.

Original: Since
Revised: Since

Original: MPT methods solve the analogical reasoning task end-to-end without any explicit relation prediction process, we use two ways to evaluate their relation-aware abilities.
Revised: MPT methods solve the analogical reasoning task end-to-end without any explicit relation prediction process, we use two ways to evaluate their relation-aware abilities.

Original: The first one is that we predict the relation via the special relation token
Revised: The first one is that we predict the relation via the special relation token

Original: [R] , which is similar to mask entities prediction and evaluate the predictions with Hit@k metrics.
Revised: [R] , which is similar to mask entities prediction and evaluate the predictions with Hit@k metrics.

Original: However, this evaluation method does not precisely reflect the relation-aware abilities of models since [R] is an abstract virtual token that may aggregate multiple relation information.
Revised: However, this evaluation method does not precisely reflect the relation-aware abilities of models since [R] is an abstract virtual token that may aggregate multiple relation information.

Original: Therefore, we devise the second method that computes the Euclidean distance as follows:
Revised: Therefore, we devise the second method that computes the Euclidean distance as follows:

Original: [Equation]
Revised: [Equation]

Original: where |S t | is the total number of the test set, h A [R] is the hidden state of [R] in the last transformer layer, E e r is the special relation embedding (described in Section 4.2.1) of the golden relation r .
Revised: where |S t | is the total number of the test set, h A [R] is the hidden state of [R] in the last transformer layer, E e r is the special relation embedding (described in Section 4.2.1) of the golden relation r .

Original: Norm ( · ) is the l 2 -normalization of vectors, d ( · , · ) is the Euclidean distance function.
Revised: Norm ( · ) is the l 2 -normalization of vectors, d ( · , · ) is the Euclidean distance function.

Original: The evaluation results are shown in Table 6, we find that MKGE methods perform better than most
Revised: The evaluation results are shown in Table 6, we find that MKGE methods perform better than most

Original: MPT methods on Hit@k metrics, especially on Hit@3 metric, which may benefit from the explicit relation perception in the pipeline process.
Revised: MPT methods on Hit@k metrics, especially on Hit@3 metric, which may benefit from the explicit relation perception in the pipeline process.

Original: Moreover, MarT FLAVA achieves the best relationaware performance on Hit@k and Euclidean distance metrics, but MarT FLAVA performs worse than MarT MKGformer in answer entity prediction as shown in Table 2.
Revised: Moreover, MarT FLAVA achieves the best relationaware performance on Hit@k and Euclidean distance metrics, but MarT FLAVA performs worse than MarT MKGformer in answer entity prediction as shown in Table 2.

Original: We speculate that the special token [R] contains not only the golden relation but also other related relation information.
Revised: We speculate that the special token [R] contains not only the golden relation but also other related relation information.

Original: C. C OMPARISON OF P ERFORMANCE AND M ODEL S IZE
Revised: C. C OMPARISON OF P ERFORMANCE AND M ODEL S IZE

Original: [Figure]
Revised: [Figure]

Original: In this section, we detail the size of MPT baseline models and compare them with their performance.
Revised: In this section, we detail the size of MPT baseline models and compare them with their performance.

Original: In MPT models, the single-stream models (VisualBERT, ViLT) are the smallest, the dual-stream models (ViLBERT) are the middle and the mixed-stream models (FLAVA, MKGformer) are the biggest.
Revised: In MPT models, the single-stream models (VisualBERT, ViLT) are the smallest, the dual-stream models (ViLBERT) are the middle and the mixed-stream models (FLAVA, MKGformer) are the biggest.

Original: The performance of models is roughly proportional to their sizes, as shown in Figure 9.
Revised: The performance of models is roughly proportional to their sizes, as shown in Figure 9.

Original: MKGformer outperforms all other models, including the biggest FLAVA model.
Revised: MKGformer outperforms all other models, including the biggest FLAVA model.

Original: D E RROR C ASE A NALYSIS
Revised: D E RROR C ASE A NALYSIS

Original: In this section, we conduct an error case study on MARS in Figure 10.
Revised: In this section, we conduct an error case study on MARS in Figure 10.

Original: From the error cases, we can see the hardship of the multimodal analogical reasoning task: 1) Imbalance of multimodal.
Revised: From the error cases, we can see the hardship of the multimodal analogical reasoning task: 1) Imbalance of multimodal.

Original: The semantic scales of images and text are inconsistent, which leads to incorrect matching (Zhu et al., 2022).
Revised: The semantic scales of images and text are inconsistent, which leads to incorrect matching (Zhu et al., 2022).

Original: Although we filter some hard-to-visualize entities in data collection in Section B.1, the high semantic entities exist.
Revised: Although we filter some hard-to-visualize entities in data collection in Section B.1, the high semantic entities exist.

Original: As shown in example (a), “management” and “control” are abstract entities that are difficult to find equivalent images.
Revised: As shown in example (a), “management” and “control” are abstract entities that are difficult to find equivalent images.

Original: Moreover, the uncoordinated convergence problem in multimodal learning further exacerbates the difficulty of the multimodal analogical reasoning task (Peng et al., 2022; Wang et al., 2020).
Revised: Moreover, the uncoordinated convergence problem in multimodal learning further exacerbates the difficulty of the multimodal analogical reasoning task (Peng et al., 2022; Wang et al., 2020).

Original: 2) One-to-more problem.
Revised: 2) One-to-more problem.

Original: It is challenging for the models to solve one-to-more entities.
Revised: It is challenging for the models to solve one-to-more entities.

Original: In example (b), “Memba” is an instance of both “snake” and “animal”, which is confusing to MKGformer.
Revised: In example (b), “Memba” is an instance of both “snake” and “animal”, which is confusing to MKGformer.

Original: 3) Unintuitive relations.
Revised: 3) Unintuitive relations.

Original: In our MARS dataset, some relations are not intuitive, requiring models to have strong relation reasoning ability.
Revised: In our MARS dataset, some relations are not intuitive, requiring models to have strong relation reasoning ability.

Original: As shown in example (c), the relation “intersection to” means the extension of the head and tail entity intersects.
Revised: As shown in example (c), the relation “intersection to” means the extension of the head and tail entity intersects.

Original: All four models are struggling and far away from the golden answer entity.
Revised: All four models are struggling and far away from the golden answer entity.

Original: watermelon ?
Revised: watermelon ?

Original: fruit instance of
Revised: fruit instance of

Original: Memba snake instance of laborer ?
Revised: Memba snake instance of laborer ?

Original: farmer control management correspond to
Revised: farmer control management correspond to

Original: MKGformer
Revised: MKGformer

Original: MKGformer*
Revised: MKGformer*

Original: TransAE
Revised: TransAE

Original: TransAE* rural body officer rural people peasant rural insect seaport rural keep alphabet correspond to
Revised: TransAE* rural body officer rural people peasant rural insect seaport rural keep alphabet correspond to

Original: Top-3 Entity
Revised: Top-3 Entity

Original: Gold Rank
Revised: Gold Rank

Original: MKGformer
Revised: MKGformer

Original: MKGformer*
Revised: MKGformer*

Original: TransAE
Revised: TransAE

Original: TransAE* zinc Warriors animal snake animal insect dinosaur traffic shipping animal plant traffic
Revised: TransAE* zinc Warriors animal snake animal insect dinosaur traffic shipping animal plant traffic

Original: Analogical Example
Revised: Analogical Example

Original: Question-Answer Pair Task Setting ?
Revised: Question-Answer Pair Task Setting ?

Original: fruit intersection to tuberculosis infectiousdiseases intersection to
Revised: fruit intersection to tuberculosis infectiousdiseases intersection to

Original: MKGformer
Revised: MKGformer

Original: MKGformer*
Revised: MKGformer*

Original: TransAE
Revised: TransAE

Original: TransAE* death extinction pain error hailstorm disease property astronaut stationery furniture habit army   963 rice
Revised: TransAE* death extinction pain error hailstorm disease property astronaut stationery furniture habit army   963 rice

Original: ID (
Revised: ID (

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Revised: ) (c)