Original: MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching
Revised: MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recently, the accuracy of image-text matching has been greatly improved bymultimodal pretrained models, all of which are trained on millions or billions ofpaired images and texts.
Revised: Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which are trained on millions or billions of paired images and texts.

Original: Different from them, this paper studies a new scenario asunpaired image-text matching, in which paired images and texts are assumed tobe unavailable during model training.
Revised: Different from them, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed to be unavailable during model training.

Original: To deal with this, we propose a simple yeteffective method namely Multimodal Aligned Conceptual Knowledge (MACK),which is inspired by the knowledge use in human brain.
Revised: To deal with this, we propose a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK), which is inspired by the knowledge use in human brain.

Original: It can be directly used asgeneral knowledge to correlate images and texts even without model training, orfurther fine-tuned based on unpaired images and texts to better generalize to certaindatasets.
Revised: It can be directly used as general knowledge to correlate images and texts even without model training, or further fine-tuned based on unpaired images and texts to better generalize to certain datasets.

Original: In addition, we extend it as a re-ranking method, which can be easilycombined with existing image-text matching models to substantially improve theirperformance.
Revised: In addition, we extend it as a re-ranking method, which can be easily combined with existing image-text matching models to substantially improve their performance.

Original:  Introduction
Revised:  Introduction

Original: Image-text matching is one of the most representative technique in the field of vision & languageunderstanding, which has wide applications such as online shopping, human-robot interaction andautonomous driving.
Revised: Image-text matching is one of the most representative techniques in the field of vision and language understanding, which has wide applications such as online shopping, human-robot interaction and autonomous driving.

Original: Its major challenge lies in how to accurately measure the cross-modal similaritybetween images and texts.
Revised: Its major challenge lies in how to accurately measure the cross-modal similarity between images and texts.

Original: Recently, by training on very large-scale (millions or billions) pairedimages and texts, various multimodal pretrained models [18, 15, 7] alleviate the challenge in asupervised learning manner.
Revised: Recently, by training on very large-scale (millions or billions) paired images and texts, various multimodal pretrained models [23, 20, 7] alleviate the challenge in a supervised learning manner.

Original: Although the accuracy of image-text matching is greatly improved,collecting and annotating such large-scale data in real world applications is time-consuming andexpensive.
Revised: Although the accuracy of image-text matching is greatly improved, collecting and annotating such large-scale data in real world applications is time-consuming and expensive.

Original: Different from them, this work attempts to study the image-text matching in the context of a newscenario, namely unpaired image-text matching , in which paired images and texts are assumed tobe unavailable during model training.
Revised: Different from them this work attempts to study the image-text matching in the context of a new scenario, namely unpaired image-text matching , in which paired images and texts are assumed to be unavailable during model training.

Original: It is motivated by the fact  the human brain can well correlatearbitrary images with texts while does not need to learn from such large-scale paired images andtexts.
Revised: It is motivated by the fact that the human brain can well correlate arbitrary images with texts while does not need to learn from such large-scale paired images and texts.

Original: Instead, it stores semantic knowledge about objects,relations, actions, attributes, etc. , which ismultimodal aligned and can be used to correlated visual and linguistic information [23, 2].
Revised: Instead, it stores semantic knowledge about objects, actions, attributes, etc. , which is multimodal aligned and can be used to correlate visual and linguistic information [28, 2].

Original: Inspiredby these neuroscience evidences, this work tries to deal with the unpaired image-text matching bymodeling humain brain-like knowledge.
Revised: Inspired by these neuroscience evidences, this work tries to deal with the unpaired image-text matching by modeling human brain-like knowledge.

Original: Unimodal visual or linguistic knowledge has been widely used for vision & language understandingsuch as image captioning [34], visual reasoning [39] and visual question answering [26].
Revised: Unimodal visual or linguistic knowledge has been widely used for vision and language understanding such as image captioning [39], visual reasoning [45] and visual question answering [31].

Original: By directlycombining these two types of knowledge together, multimodal knowledge shows more complementaryadvantages [40, 32].
Revised: By directly combining these two types of knowledge together, multimodal knowledge shows more complementary advantages [46, 37].

Original: However, existing knowledge-based methods are always designed to corporatewith paired  model training, which cannot well handle the unpaired image-text matching.
Revised: However, existing knowledge-based methods are always designed to corporate with paired data during model training, which cannot well handle the unpaired image-text matching.

Original: Anotheralternative is multimodal knowledge graphs [41], most of which are extended from existing linguisticknowledge graphs by linking semantically related images to corresponding words.
Revised: Another alternative is multimodal knowledge graphs [47], most of which are extended from existing linguistic knowledge graphs by linking semantically related images to corresponding words.

Original: However, the image-word alignment might have the following issues: 1) images usually include redundant orword-unrelated contents, so the images and words carry unequivalent semantic information, and2) word-related objects (or attributes) usually have diverse appearances in different images, sothe alignment is one-to-many rather than one-to-one.
Revised: However, the image-word alignment might have the following issues: 1) images usually include redundant or word-unrelated contents, so the images and words carry unequivalent semantic information, and 2) word-related objects (or attributes) usually have diverse appearances in different images, so the alignment is one-to-many rather than one-to-one.

Original: In fact, obtaining well-aligned multimodalknowledge is a great challenge.
Revised: In fact, obtaining well-aligned multimodal knowledge is a great challenge.

Original: In this work, we propose a new method namely Multimodal Aligned Conceptual Knowledge (MACK)for unpaired image-text matching.
Revised: In this work, we propose a new method namely Multimodal Aligned Conceptual Knowledge (MACK) for unpaired image-text matching.

Original: To remove the word-unrelated content in images, it focuses onsemantic concepts and collects pairs of words and semantically related image regions from publicavailable datasets.
Revised: To remove the word-unrelated content in images, it focuses on semantic concepts and collects pairs of words and semantically related image regions from public available datasets.

Original: Then it computes prototypical region representations and aligns them to the words,with the goal to alleviate the appearance variant.
Revised: Then it computes prototypical region representations and aligns them to the words, with the goal to alleviate the appearance variant.

Original: Based on the aligned conceptual knowledge ( i.e. ,word-prototypical region pairs), the MACK can bridge images and texts in the same feature spaceto measure their cross-modal similarities.
Revised: Based on the aligned conceptual knowledge ( i.e. , word-prototypical region pairs), the MACK can bridge images and texts in the same feature space to measure their cross-modal similarities.

Original: To make the pre-computed general knowledge better suitcertain datasets, we can further fine-tune it with the principle of region-level cycle-consistency, whichdoes not need paired images and texts.
Revised: To make the pre-computed general knowledge better suit certain datasets, we can further fine-tune it with the principle of region-level cycle-consistency, which does not need paired images and texts.

Original: Since the proposed MACK is simple yet effective, it can wellcorporate with existing image-text matching models as a re-ranking method to further improve theirperformance.
Revised: Since the proposed MACK is simple yet effective, it can well corporate with existing image-text matching models as a re-ranking method to further improve their performance.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows.

Original: 1) We study a rarely investigated problem as unpairedimage-text matching, which is under great demand in practical applications.
Revised: 1) We study a rarely investigated problem as unpaired image-text matching, which is under great demand in practical applications.

Original: We accordinglypropose a simple yet effective method to model multimodal aligned conceptual knowledge, whichcan either perform test directly or perform training based on unpaired data.
Revised: We accordingly propose a simple yet effective method to model multimodal aligned conceptual knowledge, which can either perform test directly or perform training based on unpaired data.

Original: The proposed methodcan be easily adopted by existing image-text matching models to re-rank their results to obtain betterperformance.
Revised: The proposed method can be easily adopted by existing image-text matching models to re-rank their results to obtain better performance.

Original:  Related Work 
Revised:  Related Work 2.1 Image-Text Matching

Original: 2.1 Image-Text Matching

Original: Image-text matching draws much attention recently,  many effective models trained on paired imagesand texts have been proposed.
Revised: Image-text matching draws much attention recently, and many effective models trained on paired images and texts have been proposed.

Original: Socher et al.
Revised: Socher et al.

Original: [27] and Frome et al.
Revised: [32] and Frome et al.

Original: [6] propose the early frameworkof Visual-Semantic Embedding (VSE) to correlate images and their labels.
Revised: [6] propose the early framework of Visual-Semantic Embedding (VSE) to correlate images and their labels.

Original: Kiros et al.
Revised: Kiros et al.

Original: [11] extendthe VSE from image-label matching to image-text matching, which is later improved from variousaspects such as adding intra-modal constraint [31], mining hardest negative sample [5], analyzingcanonical correlation [36], improving sentence representation [24, 14], and reasoning visual relation[17, 9].
Revised: [16] extend the VSE from image-label matching to image-text matching, which is later improved from various aspects such as adding intra-modal constraint [36], mining hardest negative sample [5], analyzing canonical correlation [41], matching local instance [11, 13], improving sentence representation [29, 19], and reasoning visual relation [22, 14].

Original: In addition to the VSE, another important framework is Cross-Modal Attention (CMA)proposed by Lee et al.
Revised: In addition to the VSE, another important framework is Cross-Modal Attention (CMA) proposed by Lee et al.

Original: [13], which uses pretrained object-level features [1] and then generatescross-modal attended representations for similarity measurement.
Revised: [18], which uses pretrained object-level features [1] and then generates cross-modal attended representations for similarity measurement.

Original: The CMA is later improved fromthe directions of cross-modal memory [8], cross-modal message passing [33], context modeling[38], iterative matching [3], and graph structure [20].
Revised: The CMA is later improved from the directions of cross-modal memory [10, 9], cross-modal message passing [38], hybrid learning [8], context modeling [44], iterative matching [3], and graph structure [25].

Original: Recently, many more complicated models[4, 15, 22, 18] based on multimodal versions of Transformer [30] are proposed, which rely on millionsor billions of paired images and texts for model training and achieve very good performance. 
Revised: Later, many more models [4, 20, 27, 23] based on multimodal versions of Transformer [35] are proposed, which rely on millions or billions of paired images and texts for model training and achieve very good performance. Recently, Huang et al.

Revised: [12] make the early attempt to study how to improve the efficiency of such large image-text matching models, which achieves promising results.

Original: Incontrast to the existing models, we attempt to study a new scenario of unpaired image-text matchingthat assumes paired images and texts are unavailable for model training, which is more challengingthan conventional image-text matching.
Revised: In contrast to the existing models, we attempt to study a new scenario of unpaired image-text matching that assumes paired images and texts are unavailable for model training, which is more challenging than conventional image-text matching.

Original: 2.2 Multimodal Knowledge
Revised: 2.2 Multimodal Knowledge

Original: There are only a few works that use multimodal knowledge for vision & language understandingtasks.
Revised: There are only a few works that use multimodal knowledge for vision and language understanding tasks.

Original: For example, Zhu et al.
Revised: For example, Zhu et al.

Original: [40] build heterogeneous graphs corresponding to visual, semanticand factual features for visual question answering.
Revised: [46] build heterogeneous graphs corresponding to visual, semantic and factual features for visual question answering.

Original: Wang et al.
Revised: Wang et al.

Original: [32] combine visual and textualknowledge to find discriminative parts of objects and improve the performance of few-shot learning.
Revised: [37] combine visual and textual knowledge to find discriminative parts of objects and improve the performance of few-shot learning.

Revised: Zhang et al.

Revised: [43] propose a concept-relation graph and then leverage it for visually grounded concept learning.

Original: In addition, different multimodal knowledge graphs [41, 21, 28] are proposed, most of which extendlinguistic knowledge graphs by aligning images to words.
Revised: In addition, different multimodal knowledge graphs [47, 26, 33] are proposed, most of which extend linguistic knowledge graphs by aligning images to words.

Original: Such cross-modal alignment could be quitenoisy because images usually include redundant or word-unrelated contents.
Revised: Such cross-modal alignment could be quite noisy because images usually include redundant or word-unrelated contents.

Original: In fact, the multimodalknowledge used in these methods is either cross-modal unaligned or noisy aligned, which usuallyneeds paired images and texts as strong supervisions for model training.
Revised: In fact, the multimodal knowledge used in these methods is either cross-modal unaligned or noisy aligned, which usually needs paired images and texts as strong supervisions for model training.

Original: Different from them, ourmultimodal aligned 
Revised: Different from them, our multimodal aligned conceptual knowledge establishes more accurate one-to-one alignment in the fine-grained concept-level, which can handle the unpaired image-text matching without using strong supervisions.

Original:  Multimodal Aligned Conceptual Knowledge
Revised:  Multimodal Aligned Conceptual Knowledge

Original: As shown in Figure1, we propose the Multimodal Aligned Conceptual Knowledge (MACK) forunpaired image-text matching.
Revised: As shown in Figure we propose the Multimodal Aligned Conceptual Knowledge (MACK) for unpaired image-text matching.

Original: To obtain the knowledge, we first collect a set of conceptual words andtheir semantically related image regions from public datasets.
Revised: To obtain the knowledge we first collect a set of conceptual words and their semantically related image regions from public datasets.

Original: For each word, obtaining a prototypicalregion representation by averaging the representations of all related regions.
Revised: For each word, we obtain a prototypical region representation by averaging the representations of all related regions.

Original: Thus, we obtain thepretrained general knowledge containing pairwise words and prototypical region representations.
Revised: Thus, we obtain the pretrained general knowledge containing pairwise words and prototypical region representations.

Original: To make the knowledge better adapt to certain datasets, we then fine-tune the pretrained generalknowledge by modeling the concept-level cycle consistency based on unpaired images and texts.
Revised: To make the knowledge better adapt to certain datasets, we then fine-tune the pretrained general knowledge by modeling the concept-level cycle consistency based on unpaired images and texts.

Original: Give an image and a text, we first perform text parsing to obtain a word graph, and use the knowledgeas a cross-modal bridge to obtain a set of prototypical region representations for all the words.
Revised: Give an image and a text, we first perform text parsing to obtain a word graph, and use the knowledge as a cross-modal bridge to obtain a set of prototypical region representations for all the words.

Original: Then,we perform object detection on the image to obtain another set of region representations.
Revised: Then, we perform object detection on the image to obtain another set of region representations.

Original: Thus, therepresentations of words and regions are in the same feature space, in which they can be directlycompared as a cross-modal matrix.
Revised: Thus, the representations of words and regions are in the same feature space, in which they can be directly compared to get a cross-modal matrix.

Original: After pooling the values in row and column dimensions based onheuristic rules, we can finally obtain the global similarity score between the image and text.
Revised: After pooling the values in row and column dimensions based on heuristic rules, we can finally obtain the global similarity score between the image and text.

Original: 3.1 Pretrained General Knowledge
Revised: 3.1 Pretrained General Knowledge

Original: The studied knowledge in this work has two major properties as follows.
Revised: The studied knowledge in this work has two major properties as follows.

Original: 1) Concept-level knowledge:the knowledge is built upon three major types of semantic concepts, which correspond to objects,attributes and actions in images, and nouns, adjectives and verbs in texts.
Revised: 1) Concept-level knowledge: the knowledge is built upon two major types of semantic concepts, which correspond to objects and attributes in images, and nouns and adjectives in texts.

Original: It is more fine-grainedthan existing multimodal knowledge graphs that roughly link global images to words.
Revised: It is more fine-grained than existing multimodal knowledge graphs that roughly link global images to words.

Original: 2) One-toone alignment: rather than align a word to multiple possible images in a one-to-many manner, theknowledge focuses on cross-modal one-to-one alignment, with the goal to alleviate the appearancevariations of objects, attributes and actions.
Revised: 2) One-to-one alignment: rather than align a word to multiple possible images in a one-to-many manner, the knowledge focuses on cross-modal one-to-one alignment, with the goal to alleviate the appearance variations of objects and attributes.

Original: We define the knowledge as a set of semantic concepts having paired multimodal representations{ w k , v k  } k =1 ,...,K , where w k ∈ N K and v k ∈ R F are the one-hot word representation and real-valuedregion representation of k -th semantic concept, respectively, and K is the total number of semanticconcepts.
Revised: We define the knowledge as a set of semantic concepts having paired multimodal representations { ( w k , v k ) } k =1 ,...,K , where w k ∈ N K and v k ∈ R F are the one-hot word representation and realvalued region representation of k -th semantic concept, respectively, and K is the total number of semantic concepts.

Original: To maximize the number of concepts, we collect all the words and their related regionsin the Visual Genome (VG) dataset [12].
Revised: To maximize the number of concepts, we collect all the words and their related regions in the Visual Genome (VG) dataset [17].

Original: We could alternatively use other datasets, but the VG has more diverse content that is useful for improving the generalization ability of the knowledge.
Revised: We could alternatively use other datasets, but the VG has more diverse content that is useful for improving the generalization ability of the knowledge.

Original: Foreach word, we can easily obtain its semantically related image regions in the dataset, and differentregions usually have diverse visual appearances.
Revised: For each word, we can easily obtain its semantically related image regions in the dataset, and different regions usually have diverse visual appearances.

Original: To avoid the impact of appearance variant, wecompute a prototypical region representation v k for each word w k by averaging all its related regionrepresentations { r j } j =1 ,...,J k :
Revised: To avoid the impact of appearance variations, we compute a prototypical region representation v k for each word w k by averaging all its related region representations { r j } j =1 ,...,J k :

Original: [Equation]
Revised: [Equation]

Original: ) where each region representation r j is obtained by feeding a bounding box and an image into thepretrained object detection model Faster-RCNN 1 [1].
Revised: ) where each region representation r j is obtained by feeding a bounding box and an image into the pretrained object detection model Faster-RCNN 1 [1].

Original: Note that the numbers of regions for differentwords { J k } i = k,...,K are quite imbalanced, which vary from tens to tens of thousands.
Revised: Note that the numbers of regions for different words { J k } k =1 ,...,K are quite imbalanced, which vary from tens to tens of thousands.

Original: Intuitively,frequently appeared words such as “man” and “dog” will have more related regions, and theircomputed prototypical region representations might be more robust than those of few-shot wordssuch as “otter” and “nun”.
Revised: Intuitively, frequently appeared words such as “man” and “dog” will have more related regions, and their corresponding prototypical region representations might be more robust than those of few-shot words such as “otter” and “nun”.

Original: However, the few-shot words can still compute initial prototypical regionrepresentations with very limited numbers ( e.g. , < 20) of regions and then refine them when morerelated regions are encountered.
Revised: However, the few-shot words can still compute initial prototypical region representations with very limited numbers ( e.g. , < 20) of regions and then refine them when more related regions are encountered.

Original: 3.2 Fine-tuned Domain Knowledge
Revised: 3.2 Fine-tuned Domain Knowledge

Original: To better adapt the pretrained general knowledge to certain datasets, we could further fine-tunethe prototypical region representations based on the principle of concept-level cycle consistency, ifunpaired images and texts in those datasets are available.
Revised: To better adapt the pretrained general knowledge to certain datasets, we could further fine-tune the prototypical region representations based on the principle of concept-level cycle consistency, if unpaired images and texts in those datasets are available.

Original: In particular, given a batch of unpairedimages and texts, we first obtain a set of detected regions using the Faster-RCNN above and a setof tokenized words.
Revised: In particular, given a batch of unpaired images and texts, we first obtain a set of detected regions using the Faster-RCNN above and a set of tokenized words.

Original: For the set of words, duplicated ones2 are removed and the rest words are{ w m } m =1 ,...,M .
Revised: For the set of words, duplicated ones are removed and the rest words are { w m } m =1 ,...,M .

Original: Then all the words can be represented by the corresponding prototypical regionrepresentations as { v m } m =1 ,...,M or V ∈ R F × M .
Revised: Then all the words can be represented by the corresponding prototypical region representations as { v m } m =1 ,...,M or V ∈ R F × M .

Original: For the set of regions, we compute similarity scoresbetween the prototypical region representations (in the pretrained general knowledge) and the set ofregion representations, and then remove the redundant regions belonging to same concepts in a similarway as Non-Maximum Suppression (NMS).
Revised: For the set of regions, we compute similarity scores between the prototypical region representations (in the pretrained general knowledge) and the set of region representations, and then remove the redundant regions belonging to same concepts in a similar way as Non-Maximum Suppression (NMS).

Original: The resulting regions are denoted as { r n } n =1 ,...,N or
Revised: The resulting regions are denoted as { r n } n =1 ,...,N or

Original: R ∈ R F × N .
Revised: R ∈ R F × N .

Original: To fine-tune the prototypical region representations to better suit for certain datasets, we design acycle consistent loss to learn a parametric transformation matrix W ∈ R F × F .
Revised: To fine-tune the prototypical region representations to better suit for certain datasets, we design a cycle consistent loss to learn a parametric transformation matrix W ∈ R F × F .

Original: The main idea is tofirst measure similarities between each word and all regions, and use the similarities as weights tocombine all regions to obtain a reconstructed word 
Revised: The main idea is to first measure similarities between each word and all regions, and use the similarities as weights to combine all regions to obtain a reconstructed word 2 :

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R M × N is the similarity matrix between transformed word and region representations,σ ( · ) is the softmax operation in the column dimension, and ˆ V ∈ R F × M contains the reconstructedword representations.
Revised: where S ∈ R M × N is the similarity matrix between transformed word and region representations, σ ( · ) is the softmax operation along the column dimension, and ˆ V ∈ R F × M contains the reconstructed word representations.

Original: Then, each original word and its reconstructed one are compared to generate alabel indicating whether they are the same or not.
Revised: Then, each original word and its reconstructed one are compared to generate a label indicating whether they are the same or not.

Original: By minimizing the cross-entropy loss L based onthe predicted label and groundtruth one, we can optimize the W :
Revised: By minimizing the cross-entropy loss L based on the predicted label and groundtruth one, we can optimize the W :

Original: [Equation]
Revised: [Equation]

Original:  where ˆ Y ∈ R M × M includes the predicted labels, in which each column is denoted as ˆ y m , and y m isthe groundtruth label in which the m -th value is one and the rest values are zeros.
Revised: (3) where ˆ Y ∈ R M × M includes the predicted labels, in which each column is denoted as ˆ y m , and y m is the groundtruth label in which the m -th value is one and the rest values are zeros.

Original: After the training,we can use the W to transform all prototypical region representations to obtain the fine-tuned domainknowledge, denoted as {  w k , u k  } k =1 ,...,K , where u k = Wv k ∈ R F .
Revised: After the training, we can use the W to transform all prototypical region representations to obtain the fine-tuned domain knowledge, denoted as { ( w k , u k ) } k =1 ,...,K , where u k = Wv k ∈ R F .

Original: 3.3 Knowledge-based Unpaired Matching
Revised: 3.3 Knowledge-based Unpaired Matching

Original: To decide whether an image and a text is matched or not, we use the knowledge 3 to measure thecross-modal similarity.
Revised: To decide whether an image and a text is matched or not, we use the knowledge 3 to measure the cross-modal similarity.

Original: For the image, we use the Faster-RCNN to obtain a set of detected regionrepresentations { r i } i =1 ,...,I or R ∈ R F × I .
Revised: For the image, we use the Faster-RCNN to obtain a set of detected region representations { r i } i =1 ,...,I or R ∈ R F × I .

Original: For the text, we use  bridge to represent all the words as a set of prototypical region representations { u j } j =1 ,...,J .
Revised: For the text, we use the knowledge as a cross-modal bridge to represent all the words as a set of prototypical region representations { u j } j =1 ,...,J .

Original: Eventhough these two set of representations are in the same feature space, they have different amount ofsemantic information, directly comparing their similarity might be sub-optimal.
Revised: Even though these two set of representations are in the same feature space, they have different amount of semantic information, directly comparing their similarity might be sub-optimal.

Original: In fact, each u j isconcept-level corresponding a word, while each r i usually describes not only an object but also itsattribute or action.
Revised: In fact, each u j is concept-level corresponding a word, while each r i usually describes not only an object but also its attribute.

Original: To make a better comparison, we infer the relation among words and aggregaterelated word representations before the similarity measurement.
Revised: To make a better comparison, we infer the relation among words and aggregate related word representations before the similarity measurement.

Original: In particular, for the words in the text, we use the Natural Language Toolkit (NLTK) 4 to predicttheir types and parse their dependencies.
Revised: In particular, for the words in the text, we use the Natural Language Toolkit (NLTK) 4 to predict their types and parse their dependencies.

Original: Then, we take each noun as an anchor and combine itsrepresentation with depended adjective representationor verb representation in an averaged manner.
Revised: Then, we take each noun as an anchor and combine its representation with dependent adjective representation in an averaged manner.

Original: Thus, the set of word representations { u j } j =1 ,...,J are re-computed as { u l } l =1 ,...,L or U ∈ R F × L .
Revised: Thus, the set of word representations { u j } j =1 ,...,J are re-computed as { u l } l =1 ,...,L or U ∈ R F × L .

Original: Then we can obtain the desired similarity score s for the image and text as:
Revised: Then we can obtain the desired similarity score s for the image and text as:

Original: [Equation]
Revised: [Equation]

Original: where ρ ( · ) is the max-mean pooling that first performs max pooling on the column dimension andthen mean pooling on the row dimension on the input matrix.
Revised: where ρ ( · ) is the max-mean pooling that first performs max pooling on the column dimension and then mean pooling on the row dimension on the input matrix.

Original: Given multiple images and texts, wecan similarly obtain their similarity scores for unpaired image-text matching.
Revised: Given multiple images and texts, we can similarly obtain their similarity scores for unpaired image-text matching.

Original: 3.4 Re-ranking Existing Models
Revised: 3.4 Re-ranking Existing Models

Original: Note that the knowledge in this work is inspired by human-like knowledge, which is quite differentfrom the  model knowledge learnt from paired data.
Revised: Note that the knowledge in this work is inspired by human-like knowledge, which is quite different from the pretrained model knowledge learnt from paired data.

Original: So the proposed MACK tends to have complementary properties when combing with existing  models.
Revised: So the proposed MACK tends to have complementary properties when combining with existing pretrained models.

Original: To demonstrate this idea, we try to use theknowledge to re-ranking the results by existing models to improve their performance.
Revised: To demonstrate this idea, we try to use the knowledge to re-rank the results by existing models to improve their performance.

Original: Taking the text-driven image retrieval as an example, given a text as query and a set of N images asgallery, an existing model can produce a similarity vector s ∈ R N × 1 .
Revised: Taking the text-driven image retrieval as an example, given a text as query and a set of G images as gallery, an existing model can produce a similarity vector s ∈ R G × 1 .

Original: By sorting all the scores of s inthe descending order, the modelusually is  able to rank the matched image in top- k .
Revised: By sorting all the scores of s in the descending order, the model is usually able to rank the matched image in top- k .

Original: Then we can tunethe top- k scores by adding the corresponding scores predicted by the knowledge with a balancingfactor λ , and then re-rank the top- k images to improve the rank of matched image. 
Revised: Then we can tune the top- k scores by adding the corresponding scores predicted by the knowledge with a balancing factor λ , and then re-rank the top- k images to improve the rank of matched image. 3.5 Discussion

Revised: Note that the proposed MACK defines a general framework for adaptive knowledge representation, organization, and reasoning.

Revised: For the knowledge representation, it uses the prototype learning to obtain pairwise prototypical region representations and semantic words.

Revised: Then, it potentially organizes the knowledge in a graph-like manner, in which the dependency relation among different knowledge pairs are explored.

Revised: At last, it hierarchically reasons the knowledge in three levels, including domain-level by fine-tuning on certain datasets, sample-level by using the pretrained models, and instance-level by performing the max-mean pooling.

Original:  Experimental Results
Revised:  Experimental Results

Original: To demonstrate the effectiveness of the proposed method, we perform extensive experiments ofimage-text matching on two publicly available datasets.
Revised: To demonstrate the effectiveness of the proposed method, we perform extensive experiments of image-text matching on two publicly available datasets.

Original: 4.1 Datasets and Protocols
Revised: 4.1 Datasets and Protocols

Original: The details of datasets and protocols are as follows.
Revised: The details of datasets and protocols are as follows.

Original: 1) Flickr30k [37] consists of 31783 imagescollected from the Flickr website.
Revised: 1) Flickr30k [42] consists of 31783 images collected from the Flickr website.

Original: Each image has 5 human annotated texts.
Revised: Each image has 5 human annotated texts.

Original: We use the publictraining, validation and testing splits, which contain 29000, 1000 and 1000 images, respectively. 2)
Revised: We use the public training, validation and testing splits, which contain 29000, 1000 and 1000 images, respectively. 2)

Revised: Details about these ablation models are explained in Section 4.3.

Original:  MSCOCO [19] consists of 123287 images, each of which is associated with 5 texts.
Revised: [Table] MSCOCO [24] consists of 123287 images, each of which is associated with 5 texts.

Original: We use thepublic training, validation and testing splits, with 113287, 1000 and 5000 images, respectively.
Revised: We use the public training, validation and testing splits, with 113287, 1000 and 5000 images, respectively

Original:  The image-text matching usually includes two sub-tasks in terms of: 1) image annotation: retrievingrelated texts given images, and 2) image retrieval: retrieving related images given texts.
Revised: [Table] The image-text matching usually includes two sub-tasks in terms of: 1) image annotation: retrieving related texts given images, and 2) image retrieval: retrieving related images given texts.

Original: Thecommonly used evaluation criterions are “R @ 1”, “R @ 5” and “R @ 10”, i.e. , recall rates at the top-1, 5and 10 results.
Revised: The commonly used evaluation criterions are “R @ 1”, “R @ 5” and “R @ 10”, i.e. , recall rates at the top-1,and 10 results.

Original: Following other methods, we also use an additional criterion of “Rs” by summing allthe recall rates to evaluate the overall performance.
Revised: Following other methods, we also use an additional criterion of “Rs” by summing all the recall rates to evaluate the overall performance.

Original: 4.2 Implementation Details
Revised: 4.2 Implementation Details

Original: In the pretrained general knowledge, we collect all the words in the Visual Genome dataset, so thetotal number of semantic concepts is K = 27801.
Revised: In the pretrained general knowledge, we collect all the words in the Visual Genome dataset, so the total number of semantic concepts is K = 27801.

Original: For each image, we use the pretrained Faster-RCNNto extract region representations, so the number of detected regions is I = 36 and the dimension of(prototypical) region representation is F = 2048.
Revised: For each image, we use the pretrained Faster-RCNN to extract region representations, so the number of detected regions is I = 36 and the dimension of (prototypical) region representation is F = 2048.

Original: In the fine-tuned domain knowledge, we performthe model training in a minibatch manner.
Revised: In the fine-tuned domain knowledge, we perform the model training in a minibatch manner.

Original: Each minibatch includes unpaired 128 images and 128texts that are randomly sampled from the whole training set.
Revised: Each minibatch includes unpaired 128 images andtexts that are randomly sampled from the whole training set.

Original: When removing the redundant regions

Original: Details about these ablation are explained in Section 4.

Original: [Table]

Original: [Table]

Original: belonging to same semantic concepts, we keep only one region that has the largest similarity scorecompared with prototypical region representations.

Original: We use the Adam algorithm [10] to optimizethe only parameter matrix W with a learning rate of 2 e − 4 for 30 epochs.
Revised: We use the Adam algorithm [15] to optimize the only parameter matrix W with a learning rate of 2 e − 4 for 30 epochs.

Original: When re-ranking existingmodels, we empirically set k = 15 and λ = 0.1.
Revised: When re-ranking existing models, we empirically set k = 15 and λ = 0.1.

Original: The knowledge-based unpaired image-text matchingis very efficient, which does not need GPUs for acceleration.
Revised: The knowledge-based unpaired image-text matching is very efficient, which does not need GPUs for acceleration.

Original: 4.3 Unpaired Image-text Matching
Revised: 4.3 Unpaired Image-text Matching

Original: Since existing models are all proposed for paired image-text matching, we compare several ablationmodels of the proposed MACK to verify their effectiveness for unpaired image-text matching.
Revised: Since existing models are all proposed for paired image-text matching, we compare several ablation models of the proposed MACK to verify their effectiveness for unpaired image-text matching.

Original: Thefirst ablation model is pretrained general knowledge, which does not need model training.
Revised: The first ablation model is pretrained general knowledge, which does not need model training.

Original: Then wedesign another three models based on it by: 1) replacing the prototypical region representation witha random region representation, similar to existing multimodal knowledge graphs, 2) removing theadjectives or verbs when re-computing the word representations, and 3) replacing the max-meanpooling with mean-mean pooling.
Revised: Then we design its three model variants including: 1) replacing the prototypical region representation with a random region representation, similar to existing multimodal knowledge graphs, 2) removing the adjectives when re-computing the word representations, and 3) replacing the max-mean pooling with mean-mean pooling.

Original: We also present another ablation model as fine-tuned domainknowledge, which additionally performs a training stage on unpaired images and texts using theconcept-level cycle consistent loss.
Revised: We also present another ablation model as fine-tuned domain knowledge, which additionally performs a training stage on unpaired images and texts using the concept-level cycle consistent loss.

Revised: We also design its two model variants including: 1) using Flickr30k images and

Revised: MSCOCO texts for the fine-tuning rather than Flickr30k images and texts, and 2) directly optimizing the cycle-consistency loss without using the knowledge.

Original: We compare the performance of unpaired image-text matching by all these models on the Flickr30kdataset in Table 1.
Revised: We compare the performance of unpaired image-text matching by all these models on the Flickr30k dataset in Table 1.

Original: It is reasonable that the pretrained general knowledge achieves much worseperformance than existing image-text matching models trained on paired images and texts.
Revised: It is reasonable that the pretrained general knowledge achieves much worse performance than existing image-text matching models trained on paired images and texts.

Original: Theunpaired image-text matching still needs more research and there is still a lot of room for performanceimprovement.
Revised: The unpaired image-text matching still needs more research and there is still a lot of room for performance improvement.

Original: By removing the region prototypes, adjectives or verbs, and max-mean pooling, theperformance all becomes worse.
Revised: By removing the region prototypes, adjectives, and max-mean pooling 5 , the performance all becomes worse.

Original: And we find that the region prototypical representations make thelargest contributions to the performance of pretrained general knowledge.
Revised: And we find that the region prototypical representation makes the largest contribution to the performance of pretrained general knowledge.

Revised: By training on the unpaired

Revised: [Table]

Original: By training on the unpairedimages and texts, the fine-tuned domain knowledge shows significant performance improvements,especially on the sub-task of image retrieval.
Revised: [Table] [Figure] images and texts , the fine-tuned domain knowledge shows significant performance improvements, especially on the sub-task of image retrieval.

Revised: By additionally removing the implicit prior that there exist one-to-one correspondences between images and texts, the results become slightly worse but are still much better than pretrained knowledge.

Revised: The results are very bad if we do not use the knowledge as the cross-modal bridge to correlate images with texts.

Original: In the following experiments, we use the pretrainedgeneral knowledge as the default MACK due to its simplicity.
Revised: In the following experiments, we use the pretrained general knowledge as the default MACK due to its simplicity.

Original: 4.4 Re-ranking State-of-the-art Models
Revised: 4.4 Re-ranking State-of-the-art Models

Original: The recent state-of-the-art models for image-text matching are variants of multimodal Transformers,all of which are pretrained on very large-scale datasets containing millions or billions of pairedimages and texts.
Revised: The recent state-of-the-art models for image-text matching are variants of multimodal Transformers, all of which are pretrained on very large-scale datasets containing millions or billions of paired images and texts.

Original: To demonstrate that our knowledge-based MACK is complementary withdatabased existing  models, we use the MACK to re-rank the retrieval results of two state-of-the-art models [Figure] Method [Table]
Revised: To demonstrate that our knowledge-driven MACK is complementary with existing data-driven models, we use the MACK to re-rank the retrieval results of two state-of-the-art models including CLIP [30] and ALBEF [21] 6 .

Original: Rs

Original: Image

Original: [Table]

Original: Image Retrieval [Table] including CLIP [25] and ALBEF [16]. The original and re-ranked performance of these two modelson the Flickr30k and MSCOCO datasets are shown in Table 2.
Revised: The original and re-ranked performance of these two models on the Flickr30k and MSCOCO datasets are shown in Table 2.

Original: It should be noted that, we use thezero-shot versions of these two models with the goal to make fair comparisons with the
Revised: It should be noted that, we use the zero-shot versions of these two models with the goal to make fair comparisons with the

Original: proposed
Revised: proposed

Original: MACK.
Revised: MACK.

Original: The zero-shot means that they are pretrained on large-scale external datasets while directlyperform test on the Flickr30k and MSCOCO datasets.
Revised: The zero-shot means that they are pretrained on large-scale external datasets while directly perform test on the Flickr30k and MSCOCO datasets.

Original: From the table, we can see that although theiroriginal accuracies are very high, re-ranking with our proposed MACK can further improve them.
Revised: From the table, we can see that although their original accuracies are very high, re-ranking with our proposed MACK can further improve them.

Original: Forthe CLIP, the re-ranking can lead to 1.4% and 1.6% improvements in R @ 1 of image annotation onthe two datasets, respectively.
Revised: For the CLIP, the re-ranking can lead to 1.4% and 1.6% improvements in R @ 1 of image annotation on the two datasets, respectively.

Original: For the ALBEF, the re-ranking can lead to overall 6.4% improvementin Rs on the Flickr30k dataset.
Revised: For the ALBEF, the re-ranking can lead to overall 6.4% improvement in Rs on the Flickr30k dataset.

Original: These evidences demonstrate that the MACK does show complementary properties when combingwith existing models.
Revised: These evidences demonstrate that the MACK does show complementary properties when combining with existing models.

Original: To further explain this, we show some examples of retrieved images based ontext queries by the CLIP in Figure 2.
Revised: To further explain this, we show some examples of retrieved images based on text queries by the CLIP in Figure 2.

Original: These examples are selected based on the following rules: 1)ranks of groundtruth matched images (marked by red bounding boxes in the figure) by the CLIP arenot top-1, and 2) our proposed MACK can re-rank these matched images higher.
Revised: These examples are selected based on the following rules: 1) ranks of groundtruth matched images (marked by red bounding boxes in the figure) by the CLIP are not top-1, and 2) our proposed MACK can re-rank these matched images higher.

Original: In the first example,it seems that the CLIP cannot well understand the “yellow hat”, “red apron” or “pink shirt” in thequery text, since the first retrieved image does not contain either of these semantic concepts but rankshigher than the groundtruth one.
Revised: In the first example, it seems that the CLIP cannot well understand the “yellow hat”, “red apron” or “pink shirt” in the query text, since the first retrieved image does not contain either of these semantic concepts but ranks [Table] higher than the groundtruth one.

Original: While our MACK especially focuses on these semantic conceptsand can improve the corresponding similarity between the matched image and query text.
Revised: While our MACK especially focuses on these semantic concepts and can improve the corresponding similarity between the matched image and query text

Revised: [Table]

Original: 4.5 Cross-dataset Image-text Matching
Revised: 4.5 Cross-dataset Image-text Matching

Original: We also try to re-rank existing image-text matching models when applying them to the scenario ofcross-dataset image-text matching.
Revised: We also try to re-rank existing image-text matching models when applying them to the scenario of cross-dataset image-text matching.

Original: The experimental settings are explained as follows: 1) selectingtwo representative image-text matching models (VSRN [25] and SAEM [35]) and training them on asource dataset ( e.g. , Flickr30k or MSCOCO), 2) performing model test on a different target dataset( e.g. , MSCOCO or Flickr30k), and 3) using the proposed MACK to re-rank their results on the targetdataset.
Revised: The experimental settings are explained as follows: 1) selecting two representative image-text matching models (VSRN [30] and SAEM [40]) and training them on a source dataset ( e.g. , Flickr30k or MSCOCO), 2) performing model test on a different target dataset ( e.g. , MSCOCO or Flickr30k), and 3) using the proposed MACK to re-rank their results on the target dataset.

Original: Note that either  the two models or our MACK does not use the target dataset for modeltraining.
Revised: Note that either our MACK or the two models do not use the target dataset for model training.

Original: In Table 3, the results of two directions of cross-dataset generalization, i.e. , MSCOCO -> Flickr30kand Flickr30k -> MSCOCO, are both presented.
Revised: In Table 3, the results of two directions of cross-dataset generalization, i.e. , MSCOCO -> Flickr30k and Flickr30k -> MSCOCO, are both presented.

Original: We can find that using the MACK to re-rank the
Revised: We can find that using the MACK to re-rank the

Original: VSRN and SAEM can improve their generalization abilities in the cross-dataset scenario.
Revised: VSRN and SAEM can improve their generalization abilities in the cross-dataset scenario.

Original: Although[Table] the relative improvements might be a little bit small, they are substantial in all 7 evaluation criterions.
Revised: Although the relative improvements might be a little bit small, they are substantial in all 7 evaluation criterions.

Original: Comparing Table 3 with Table [Table] it seems that the performance improvements of re-ranking partiallydepend on the performance of base models.
Revised: Comparing Table 3 with Table 2, it seems that the performance improvements of re-ranking partially depend on the performance of base models.

Original: When the base models are more accurate, the performanceimprovements of re-ranking are larger.
Revised: When the base models are more accurate, the performance improvements of re-ranking are larger.

Original: 4.6 Prototypical Representation Analysis
Revised: 4.6 Prototypical Representation Analysis

Revised: [Figure]

Original: In the proposed MACK, one of the most important part is the prototypical region representation.
Revised: In the proposed MACK, one of the most important part is the prototypical region representation.

Original: Even the same object could have completely different appearances in images, so using how manyregions to compute each region prototype is crucial.
Revised: Even the same object could have completely different appearances in images, so using how many regions to compute each region prototype is crucial.

Original: We count the number of regions for each wordin the Visual Genome dataset and show the statistical result in Figure 3, in which the horizontal axisis word index and vertical axis is logarithmic transformed number of regions.
Revised: We count the number of regions for each word in the Visual Genome dataset and show the statistical result in Figure 3, in which the horizontal axis is word index and vertical axis is logarithmic transformed number of regions.

Original: From the figure, wecan see that about 80% words have very small numbers of regions, i.e. , less than 20 (denoted by thedashed line).
Revised: From the figure, we can see that about 80% words have very small numbers of regions, i.e. , less than 20 (denoted by the dashed line).

Original: Therefore, for most semantic concepts in the knowledge, computing their prototypicalregion representations is data-efficient.
Revised: Therefore, for most semantic concepts in the knowledge, computing their prototypical region representations is data-efficient.

Original: [Figure]

Original: To verify whether using small numbers of regions has large impact on the performance, wedefine a threshold f that controls the maximum number of regions.
Revised: To verify whether using small numbers of regions has large impact on the performance, we define a threshold f that controls the maximum number of regions.

Original: We use the ALBEF + MACK as the base model and select f from {20,50, 100, ∞ }, where ∞ is the default setting of ALBEF + MACK.
Revised: We use the ALBEF + MACK as the base model and select f from {20, 50, 100, ∞ }, where ∞ is the default setting of ALBEF + MACK.

Original: The experimental results onthe Flickr30k and MSCOCO datasets are shownin Table 4.
Revised: The experimental results on the Flickr30k and MSCOCO datasets are shown in Table 4, in which p is the percentage of how many word-region annotations of Visual Genome dataset are used.

Original: We can see that as the f becomeslarger the performance becomes slightly better.
Revised: We can see that as the f becomes larger [Table] [Table] the performance becomes slightly better.

Original: It seems that different numbers of regions do nothave a significant impact on the performance.
Revised: It seems that different numbers of regions do not have a significant impact on the performance.

Original: Itis reasonable since computing the mean mightnot need too many samples.
Revised: It is reasonable since computing the mean might not need too many samples.

Original: This is also a goodproperty for the knowledge, since if we want toadd a new semantic concept, we only have toannotate a limited number of regions.
Revised: This is also a good property for the knowledge, since if we want to add a new semantic concept, we only have to annotate a limited number of regions.

Revised: 4.7 Hyperparameter Analysis

Revised: In Table 5, we perform the experiment of image-text matching by re-ranking CLIP using different k and λ on the Flickr30k dataset, to study the impact of these two hyperparameters on the final performance.

Revised: In the table, k is the length of re-ranked images/texts, and λ is the balancing factor between knowledge and CLIP.

Revised: In the table, we vary k ∈ { 10 , 15 , 20 } and λ ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , and find that the final performance is not very sensitive to the hyperparameters.

Original: 4.7 Knowledge Visualization
Revised: 4.8 Knowledge Visualization

Original: At last, we visualize the pretrained general knowledge and fine-tuned domain knowledge (trainedon the Flickr30k dataset) in Figure 4, to illustratethe major change after the training.
Revised: At last, we visualize the pretrained general knowledge and fine-tuned domain knowledge (trained on the Flickr30k dataset) in Figure 4, to illustrate major changes after the training.

Original: In the figure,there are two low-dimensional word distributions (surrounded by solid boxes) corresponding tothe pretrained general knowledge and fine-tuned domain knowledge, respectively.
Revised: In the figure, there are two low-dimensional word distributions (surrounded by solid boxes) corresponding to the pretrained general knowledge and fine-tuned domain knowledge, respectively.

Original: The two worddistributions are obtained by using the t-SNE method [29] to embed two set of prototypical regionpresentations ( i.e. , { v k } or { u k } , k = 1,...,K).
Revised: The two word distributions are obtained by using the t-SNE method [34] to embed two set of prototypical region presentations ( i.e. , { v k } or { u k } , k = 1,...,K).

Original: For clear illustration, we set K = 100 and only selectthe most frequently used 100 semantic concepts in the Flickr30k dataset.
Revised: For clear illustration, we set K = 100 and only select the most frequently used 100 semantic concepts in the Flickr30k dataset.

Original: For each semantic concept,we show the corresponding word in the two-dimensional coordinate.
Revised: For each semantic concept, we show the corresponding word in the two-dimensional coordinate.

Original: We also group semanticallyrelated words and show them with different colors.
Revised: We also group semantically related words and show them with different colors.

Original: [Figure]

Original: [Figure]

Original: In the two word distributions, we group semantically related words and represent them using same colors.

Original: We also mark three representative groups by dashed lines, which contain words about animals, indoor objects and transports, respectively.

Original: In the left word distribution, we can see that semantically related words stay nearby.
Revised: In the left word distribution, we can see that semantically related words stay nearby.

Original: For example, thewords in the three groups (marked by dash lines) are about animals, indoor objects and transports,respectively.
Revised: For example, the words in the three groups (marked by dash lines) are about animals, indoor objects and transports, respectively.

Original: In the center area, there are many words (marked by the green color) about colors.
Revised: In the center area, there are many words (marked by the green color) about colors.

Original: However, there are still some related words that have remote distances.
Revised: However, there are still some related words that have remote distances.

Original: For example, in the bottomthere are also several words about animals such as bird, dog, bear and cat, which are far from thewords in the first group.
Revised: For example, in the bottom there are also several words about animals such as bird, dog, bear and cat, which are far from the words in the first group.

Original: After fine-tuning the knowledge with unpaired data, we find the worddistribution is semantically more compact.
Revised: After fine-tuning the knowledge with unpaired data, we find the word distribution is semantically more compact.

Original: We can see that all the words about animals are very closestaying together in the first group.
Revised: We can see that all the words about animals are very close staying together in the first group.

Original: In the second group, more words about transports such as planeand boat are added.
Revised: In the second group, more words about transports such as plane and boat are added.

Original: These evidences indicate that by training with the concept-level cycle consistency,related semantic concepts are able to leverage their potential relations to make the prototypical regionrepresentations more discriminative. 
Revised: These evidences indicate that by training with the concept-level cycle consistency, related semantic concepts are able to leverage their potential relations to make the prototypical region representations more discriminative. [Figure]

Revised: In the two word distributions, we divide words into semantically related groups and represent them using different colors.

Revised: We also mark three representative groups by dashed lines, which contain words about animals, indoor objects and transports, respectively.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: This work have studied a practically important but seldom investigated problem as unpaired imagetext matching, in which paired images and texts are not available during model training.
Revised: This work has studied a practically important but seldom investigated problem as unpaired image-text matching, in which paired images and texts are not available during model training.

Original: To deal withthis problem, inspired by the knowledge use in human brain, we have proposed a simple yet effectivemethod namely Multimodal Aligned Conceptual Knowledge (MACK).
Revised: To deal with this problem, inspired by the knowledge used in human brain, we have proposed a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK).

Original: Compared with existingmultimodal knowledge methods, the MACK has two majors properties in terms of conceptual leveland one-to-one alignment.
Revised: Compared with existing multimodal knowledge methods, the MACK has two major properties in terms of conceptual level and one-to-one alignment.

Original: It can either perform test directly or perform training based on unpaireddata, as well as re-rank existing image-text matching models to improve their performance.
Revised: It can either perform test directly or perform training based on unpaired data, as well as re-rank existing image-text matching models to improve their performance.

Original: We havedemonstrated its effectiveness with extensive experiments.
Revised: We have demonstrated its effectiveness with extensive experiments.

Original: To the best of our knowledge, this might be the first work to study the unpaired image-text matching.
Revised: To the best of our knowledge, this might be the first work to study the unpaired image-text matching.

Original: So the goal of this initial work is not pursuing high performance but to explore the modeling ofknowledge.
Revised: So the goal of this initial work is not pursuing high performance but to explore the modeling of knowledge.

Original: In the future, we will study  effective strategies to improve the performance.
Revised: In the future, we will study more effective strategies to improve the relatively low accuracy.

Original: In addition,we will also consider to mine more useful information from the knowledge from the aspects ofgraphic knowledge structure, dynamic knowledge updating and advanced knowledge reasoning.Checklist
Revised: In addition, we will also consider to mine more useful information from the knowledge from the aspects of dynamic knowledge updating and advanced knowledge reasoning.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: Acknowledgements

Original: [Yes] See Section 5.(c) Did you discuss any potential negative societal impacts of your work? [No] This work does nothave potential negative societal impacts. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: This work was jointly supported by National Key Research and Development Program of China

Original: [Yes]2.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: In the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Sections 4.1 and 4.2.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?

Original: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,

Original: internal cluster, or cloud provider)?

Original: [Yes] See Section 4.2. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.(b)
Revised: Grant No. 2018AAA0100400, National Natural Science Foundation of China (62236010, 62276261, 61721004, and U1803261), Key Research Program of Frontier Sciences CAS Grant No. ZDBS-LYJSC032, Beijing Nova Program (Z201100006820079), and CAS-AIR.

Original: Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]