Fine-Grained Matching with Multi-Perspective Similarity Modeling for Cross-Modal Retrieval
Key of Multi-source Information Mining
Abstract
Cross-modal retrieval relies on learning intermodal correspondences.
Most existing approaches focus on learning global or local correspondence and fail to explore ﬁne-grained multi-level alignments.
Moreover, it remains to be investigated how to infer more accurate similarity scores.
In this paper, we propose a novel ﬁne-grained matching with
Multi-Perspective Similarity Modeling (MPSM)
Network for cross-modal retrieval.
Speciﬁcally, the
Knowledge Graph Iterative Dissemination (KGID) module is designed to iteratively broadcast global semantic knowledge, enabling domain information to be integrated and relevant nodes to be associated, resulting in ﬁne-grained modality representations.
Subsequently, vector-based similarity representations are learned from multiple perspectives to model multi-level alignments comprehensively.
The Relation Graph Reconstruction (SRGR) module is further developed to enhance cross-modal correspondence by constructing similarity relation graphs and adaptively reconstructing them.
Extensive experiments on the Flickr30K and MSCOCO datasets validate that our model signiﬁcantly outperforms several state-of-the-art baselines.
INTRODUCTION
Cross-modal retrieval refers to retrieving the most relevant text (image) by utilizing the image (text) as a query.
Its core is to detect the potential correlation between different modalities and then measure cross-modal similarity to achieve relatively accurate matching [Hou et al., 2021].
Existing methods mainly learn global or local alignment between image and text for retrieval.
The global alignment
[Figure]
alignment.
learning methods [Hardoon et al., 2004, Karpathy and Li, 2015, Zheng et al., 2020] aim to discover correspondences between entire image and text, but fail to investigate ﬁnegrained correspondence between regions and words.
As a result, local alignment learning methods [Niu et al., 2017,
Chen et al., 2019, Chen and Luo, 2020] are presented that are used to model the region-word correspondence.
However, global or local alignment is one-sided.
Therefore, some researchers jointly model global and local correspondences to obtain multi-level alignments.
The multi-level alignment methods [Huang et al., 2018a, Li et al., 2020, Nguyen et al., 2021, Yuan et al., 2021] can provide more complimentary as well as comprehensive semantic cues, thus improving performance.
Furthermore, ﬁne-grained alignment should account for both inter-modal and intra-modal correlations.
More importantly, similarity representation and learning are key to cross-modal matching.
Most approaches [Chen et al., 2020, Wang et al., 2020, Li et al., 2019] compute scalarbased cosine distances to reﬂect the cross-modal similarity.
However, this merely yields a constant scalar value and fails to adaptively reﬁne the visual-semantic correspondence.
Recently, some novel methods [Kuang et al., 2019, Diao et al., 2021] have achieved excellent results by representing similarity with vectors rather than scalars.
These methods, however, use a single form of similarity representation and fail to learn cross-modal similarity in a ﬁne-grained manner.
In summary, there are several challenges for ﬁne-grained image-text retrieval.
To begin, it needs to consider both global and local alignment, as they facilitate the interaction between the "object" and the "global context", as shown in
Figure 1, where the region-global interaction "women on a tennis court".
Second, existing methods fail to explore ﬁnegrained intra-modal correlation which can provide richer semantic information.
As illustrated in Figure 1(b), the interaction between the "racket", the "ball", and the women’s "arm" regions corresponds to the phrase "playing tennis".
Signiﬁcantly, the representation of similarity and its learning should be considered from multiple perspectives.
For example, "A woman is hitting a tennis."
is semantically related to the image in Figure 1(vectors are in the same direction), yet they are mismatched (there are numerical differences).
Motivated by these, we propose a novel ﬁne-grained matching with Multi-Perspective Similarity Modeling (MPSM)
Network for cross-modal retrieval.
Speciﬁcally, we ﬁrst construct visual and textsual semantic knowledge graphs.
Then, we introduce the Knowledge Graph Iterative Dissemination (KGID) module that learns ﬁne-grained intra-modal correlations and modal representations by iteratively propagating the knowledge.
Subsequently, we learn vector-based similarity representations from multiple perspectives separately to model multi-level alignment.
The proposed similarity representations are learnable and can comprehensively explore image-text correspondences.
Furthermore, we designed the
Similarity-Relation Graph Reconstruction (SRGR) module to achieve more accurate matching by constructing similarity relation graphs and adaptively reconstructing similar relations.
Our main contributions are summarized below:
• We propose a KGID module that integrates domain information of nodes and iteratively propagates semantic information to neighboring nodes to capture ﬁne-grained local and global representations.
• We learn vector-based similarity from multiple perspectives, which allows for more comprehensive learning of multi-level correspondences.
To our knowledge, this is the ﬁrst work to considers both distance and direction similarity for similarity representation and learning.
• We design a SRGR module in which the similarity relational graphs are constructed and reconstructed adaptively to achieve information interaction between multi-level alignments, ﬁlter interference and enhance similarity, thus improve matching accuracy.
RELATED WORK 2.1 IMAGE-TEXT RETRIEVAL
Existing methods can be roughly split into global alignment, local alignment and multi-level alignment learning methods.
The global alignment learning methods seek to learn correspondences between the entire image and text.
Frome et al.
[2013] were the ﬁrst to map the full image and text into a common space for semantic alignment.
Some approaches are inspired by generative adversarial network (GAN)[Goodfellow et al., 2014], Wang et al.
[2017] employs GAN to produce features.
There were also methods focusing on optimization, Faghri et al.
[2018] presented an optimization scheme that increases the distance between samples and hard samples.
Furthermore, Wang et al.
[2018] emphasized the need of maintaining both inter-modal and intra-modal correspondence.
Nevertheless, the above mothods neglected ﬁne-grained semantic associations between regions and words, as well as intra-modal associations.
The local alignment learning methods explore region-word correspondence to acquire more accurate similarities.
Karpathy et al.
[2014] made the ﬁrst attempt by combining the alignment of related region-word pairings.
Lee et al.
[2018] used an attention mechanism to align each region with all words, verifying the efﬁciency of region-word alignment.
Many of the following works were based on [Lee et al., 2018].
For example, Wang et al.
[2019b] followed [Lee et al., 2018] to model the region-word relation.
Several motheds focused on both inter-modal and intra-modal relations, such as [Liu et al., 2019, Zhang et al.,
Wei et al., 2020].
However, these mothods failed to comprehensively explore ﬁne-grained visual-semantic similarity.
Unlike, we dynamically explore intra-modal correlations and model multi-level correspondences for more complete alignment.
Recently, researchers are increasingly exploring both global correspondence and local correspondence to measure crossmodal similarity comprehensively[Qi et al., 2018, Ma et al., 2019, Wen et al., 2020].
Some approaches ﬁrst tried [Qi et al., 2018, Ma et al., 2019] to tackle the image-text matching in a multi-pathway, computing the global and local similarities, and combining them into the ﬁnal similarity.
However, these approaches ignore that a word or a region may have different semantics in different global contexts, while global contextual information can be used as a clue for semantically similar samples [Wei et al., 2021, Xian et al., 2022].
Based on this, Ji et al.
[2021] implemented localto-local, global-to-local, and global-to-global cross-modal alignments in turn.
Further, Diao et al.
[2021] computed similarity representations for all local and global representation pairs simultaneously.
However, existing multi-level alignment methods were insufﬁcient for the learning of similarity, capturing only limited information.
2.2 SIMILARITY REPRESENTATION LEARNING
The core of cross-modal retrieval is the learning of similarity between different modalities.
As for global alignment methods, some [Faghri et al., 2018, Wang et al., 2016, Li et al., 2021] explored similarity by computing the cosine distance.
Another branch [Vendrov et al., 2015, Huang et al., 2018b,
Gu et al., 2018] introduced ordered representations.
As for local alignment, most methods [Liu et al., 2019, Wang et al., 2019b] computed scalar-based cosine distance to reﬂect region-word similarity.
Furthermore, most multi-level alignment methods [Qi et al., 2018, Ma et al., 2019] modeled local and global alignments separately by using scalar-based cosine distance and combined them to reﬂect the similarity.
The above approaches’ similarity representations are scalar values that cannot learn ﬁne-grained visual-semantic correlations adaptively.
Recently, some innovative approaches [Diao et al., 2021] to similarity representation and learning have been developed, Diao et al.
[2021] learned vector-based similarity representation to explore multi-level alignment, and achieved some improvement.
However, for similarity, it appears to be one-sided and fails to learn the correlation between vectors in a thorough manner.
Differently, we learns vector-based similarities from multiple perspectives to model multilevel alignment more comprehensively.
2.3 THE DIFFERENCE WITH OTHER METHOD
Compared to the GSMN [Liu et al., 2020], which also captures semantic information, while it lacks the mining of ﬁne-grained intra-modal interactions.
Instead of conducting basic matching, we consider integrating and spreading semantic knowledge among nodes.
It allows for the dynamic mining of intra-modal correlations as well as the capturing of semantically rich features.
Besides, comparison to
SGRAF [Diao et al., 2021], which also utilizes graph reasoning.
One of the key differences is that we not only mine the rich semantic information within the modality but also adaptively associates relevant nodes.
It thus enables semantically rich intra-modal correlations to be included in inter-modal similarity learning and inference, which SGRAF [Diao et al., 2021] does not make possible.
Another aspect, we cleverly model multi-level alignments and perform similarity inference from multiple perspectives, whereas most methods simply analyze a single angle.
To our knowledge, it is the ﬁrst study to consider vector-based similarity representations and learning from multiple perspectives that are complementary.
It enables more comprehensive learning of cross-modal correspondences.
METHODOLOGY
As shown in Figure 2, we ﬁrst construct the semantic knowledge graphs.
The KGID modules are then developed to learn ﬁne-grained modal representations.
Subsequently, vectorbased similarity representations are learned from two perspectives: distance or direction similarity, to comprehensively explore multi-level correspondences.
Finally, the
SRGR module, which promotes the interaction between global and local alignments by constructing and adaptively reconstructing similarity relation graphs.
3.1 SEMANTIC KNOWLEDGE GRAPH CONSTRUCTION 3.1.1 Visual Semantic Knowledge Graph
Given a raw image I , we use the Faster-RCNN [Krishna et al., 2017], which is pre-trained on Visual Genome, to detect n ( n = 36 ) prominent regions.
Then, we add a fully connected layer to transform them into D-dimensional vectors to obtain region representations V = { v 1 , v 2 , . .
. , v n } .
Formally, we aim to create an undirected, fully connected visual semantic knowledge graph G v =
{ V v , E v } for each image, with the detected regions set as nodes, and the edge denoted as a matrix E v .
On the one hand, there are spatial dependencies between regions.
For example, "people on the court" and "people outside the court" reﬂect the spatial location relationship between "people" and "court".
Thus, we follow [Norcliffe-Brown et al., 2018] in modeling spatial dependencies between regions using polar coordinates and representing them as a spatial dependence matrix P v .
On the other hand, there are also potential semantic correlations between regions.
For example, the semantic information "playing tennis" is formed by associating the three regions: "women", "racket" and "tennis".
Therefore, we calculate the semantic correlation matrix r v between regions:
[Equation]
where λ is the scale factor.
r vij denotes the correlation between the i -th region and the j -th region.
The visual semantic knowledge graph is made up of the spatial interdependence and the semantic correlations between regions.
We calculate the Hadamard product of r v and P v , then apply column L2-normalization to obtain the edge matrix E v .
[Equation]
where (cid:107)·(cid:107) 2 denotes column L2-normalization.
3.1.2 Textual Semantic Knowledge Graph
For a text T comprising m words, we ﬁrst represent each word as a continuous embedding vector.
Then the word vectors are embedded into a bi-directional GRU [Cho et al., 2014].
Finally, textual word feature representation is obtained, denoted as T = { t 1 , t 2 , . . .
, t m } .
To construct a textual semantic knowledge graph G t = { V t , E t } , we set words as nodes, which are semantically related to each other.
To obtain the syntactic dependency matrix P t , we ﬁrst utilize Stanford CoreNLP to ﬁnd syntactic
Semantic Knowledge Graph Construction
[Figure]
Graph
Multi-Perspective Similarity
Representation j t i v j t
i v
KGID
Global Similarity Representation
Relational Graph Reconstruction
Local Similarity Representation twowomenona tennis court playingtennis {𝑣 𝑖 } {෤𝑣 𝑖 } { ҧ𝑣 𝑖 } 𝑐 𝑡 {𝑡 𝑗 } 𝑃 𝑡 𝑐 𝑣 𝑃 𝑣 { ǁ𝑡 𝑗 } { ҧ 𝑡 𝑗 } 𝑑 𝑗𝑙 or {𝑐 𝑗𝑙 } 𝑑 𝑔 or 𝑐 𝑔 fc dependencies, and add self-loops (the matrix diagonal is 1).
We compute the correlation r t between words.
Similarly, the E t is then obtained by performing a column l2-normalizatio operation on the Hadamard product of r t and P t :
[Equation]
3.2 KNOWLEDGE GRAPH ITERATIVE DISSEMINATION
Considering that both images and texts are based on the
KGID module for knowledge propagation, we ﬁrst depict the knowledge propagation process on the visual semantic knowledge graph in detail, and then roughly on the textual.
3.2.1 Visual KGID
Given a visual semantic knowledge graph G v = { V v , E v } , its node representation denoted as V .
We use the associated edges to associate each node with other nodes and propagate knowledge, resulting in a new visual semantic association feature map.
Then, the softmax function is used to learn each region’s knowledge weight coefﬁcients and update the nodes’ own features by element multiplication.
Therefore, we obtain the the "knowledgeable" local representation by
[Equation]
where A 0 =
E v , W ( l ) ∈ R d × d , ρ is the softmax function.
In order to aggregate and enhance semantic correlations, we design an adaptive gate mechanism with fusion and reconstruction procedures in the process of knowledge propagation and reasoning.
We obtain gating mask by combining
V ( l ) k and V .
Then the gating mask are utilized to control the ﬂow of information between V ( l ) k and V .
Therefore the image feature representation is dynamically updated by
[Equation]
where σ is the sigmoid function, (cid:12) is the Hadamard product.
Finally, we perform aggregated inference followed by shortcut connection to enhance the local feature representation:
[Equation]
where W ( l ) kg ∈ R d × d .
Moreover, we follow [Kuang et al., 2019] to update the edge A ( l +1) using the afﬁnity of new nodes, i.e., by adaptively update the semantic knowledge through the afﬁnity between regions:
[Equation]
where W ( l +1)
I ∈ R
d × d and W ( l +1)
O ∈ R d × d are linear transformations of incoming and outgoing nodes, respectively.
A ( l +1) means that if two regions are highly correlated, edges with higher scores will connect the nodes.
We iteratively infer, update, and aggregate the visual semantic knowledge graph in N steps, and use the output of the last step as the local inference representation ˜ v of the image.
To obtain the global feature, we perform a self-attentive operation on the local region features.
Speciﬁcally, given the mean-pooled feature v m
[Equation]
with v m as the query, and ˜ v as the key and value, we ﬁrst calculate the attention weight distribution a v = softmax ( W v 1 (( W v 2 ˜ v ) (cid:12) ( W v 3 v m ))) for all regions, where W v 1 ∈ R 1 × d , W v 2 ∈ R n × d and W v 3 ∈ R n × d .
Then, the image global feature is obtained by ¯ v = 1 N (cid:80) Ni
=1 a vi ˜ v i .
3.2.2 Textual KGID
Similarly, given a textsual semantic knowledge graph G t = { V t , E t } , its node representation is denoted as T , we perform an N -step iterative propagation and inference to mine ﬁne-grained local features ˜ t .
Subsequently, a self-attentive operation is performed to mine the global features ¯ t .
3.3 MULTI-PERSPECTIVE SIMILARITY REPRESENTATION LEARNING
We learn local and global similarity representations by using distance or directional differences between vectors.
It enables a more comprehensive understanding of similarity.
In order to enhance the visual representation, we use the words information from each sentence as cues to focus on all regions in each image.
For each image, we ﬁrst compute the cosine similarity between each region and each word to establish the relationship R .
The softmax function is then used to calculate the attention weights.
Finally, we construct the augmented representation of images associated with the j -th word by v tj = (cid:80) mi =1 softmax ( R ij )
˜ v i .
3.3.1 Distance Similarity
To learn the distance similarity between vectors, we ﬁrst calculate the vector-based squared Euclidean distance between vectors x ∈ R d and y ∈ R d by dis ( x , y )
=
( x − y ) 2 .
Then, the distance similarity function denotes as
[Equation]
[Equation]
where W d ∈ R m × d is a learnable parameter matrix.
Using the Eq.(10), we compute the local similarity representation d lj between v tj and ˜ t
j , then calculate the global similarity representation d g between ¯ v and ¯ t :
[Equation]
3.3.2 Direction Similarity
We consider the similarity representation learning based on the cosine distance between x ∈ R d and y ∈ R d .
Thus, the "direction" similarity representation is deﬁned as
[Equation]
where dir ( x , y ) =
( x · y ) /
( (cid:107) x (cid:107) · (cid:107) y (cid:107) ) .
We calculate the local similarity representation c lj between feature v tj and ˜ t j with Eq.(13), and calculate the global similarity representation c g between ¯ v and ¯ t :
[Equation]
3.4 SIMILARITY-RELATIONAL GRAPH RECONSTRUCTION 3.4.1 Relational Graph Building
Formally, we construct a directed relational weighted graph of similarity representations.
Speciﬁcally, we denote all "distance" (or "direction") similarity representations as graph nodes N = (cid:8) s g , s l 1 , s l 2 , ..., s l j (cid:9) , where s g denotes d g (or c g ), s lj denotes d lj (or c lj ).
For any node, the relationship between nodes is extracted from node s a to node s b and is deﬁned as a variable edge weight by [Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] where W out ∈ R m × 1 and W in ∈ R 1 × m are the linear transformations of outgoing and incoming nodes, the " ⊕ " indicates concatenation.
R ( s a , s b ) =
BN ( W out s a ) ⊕ ( BN ( W in s b )) is the trend score of the node-node relationship, and the edge weights E ( s a ,s b ) ( W out , W in ) can be calculated by using sigmoid function σ .
Note that s a → s b differs from s a ← s b , i.e. the edges are directed.
3.4.2 Relational Graph Reconstruction
We perform a series of processes, such as propagation of similarity relations and gate mechanisms, to achieve the interaction of similar information and the reconstruction of the similarity-relational graph.
Since the relational edges are directed, we take the outgoing and incoming inference, respectively, to implement bi-SRGR:
[Equation]
where W out ∈ R m × m and W in ∈ R m × m
The edge weights of s a output and input are denoted by −−−−−→
E ( s a ,s b ) ( W out , W in ) and ←−−−−− E ( s a ,s b ) ( W in , W out ) , respectively.
−→ ˜ s a and ←− ˜ s a denote the results of propagating all similarity information from node s a outgoing and incoming, respectively, both of which contain the same node s a .
Furthermore, to improve the quality of dynamic decisionmaking, we propose a conditional selection strategy to adaptively ﬁlter node information and suppress unnecessary information.
Speciﬁcally, −→ ˜ s a and ←− ˜ s
a are ﬁrst concatenated, followed by a fully connected layer and a sigmoid function to obtain the conditional mask,
[Equation]
Then, we use the generated conditional control masks to control the information ﬂow of the original −→ ˜ s a and ←− ˜ s a , followed by a shortcut connection to achieve an adaptively ﬁltered and enhanced similarity representation, thus the reconstructed −→ ˜ s ∗ a and ←− ˜ s ∗ a can be achieved by
[Equation]
Furthermore, we aggregate −→ ˜ s ∗ a and ←− ˜ s ∗ a , followed by a fully connected layer, n, which is formulated as,
[Equation] [Equation]
where W ∗ ∈
R m × m .
Finally, we feed s ∗ a into a fully connected layer to predict the ﬁnal similarity score.
3.5 TRAINING OBJECTIVES AND INFERENCE STRATEGIES
We employ bidirectional triplet ranking loss as the objective function.
Given a representation of the matched image-text pair ( v , t ) , its corresponding negative pairs are denoted as ( t , v − ) and ( v , t − ) .
We compute the loss with
L dis ( v , t ) = (cid:80) ( v , t ) { max [0 , γ − S dis ( v , t ) + S dis ( v , t − )]
+ max [0 , γ − S dis ( v , t ) + S dis ( v − , t )] } .
(21) where S dis ( v , t ) is the similarity prediction function based on the "distance" similarity representation.
Similarly, we deﬁne the ranking loss of MPSM(dir) as L dir .
In this paper, we use the proposed "distance" and "direction" similarity representations to investigate two training and inference strategies: joint training and independent training.
For joint training, We combine L dis and L dir to train our
MPSM model, i.e., we combine the "distance" and the "direction" similarity representation for training.
For independent training, we train two single model, MPSM (dis) based on the "distance" similarity representation and MPSM (dir) based on the "direction" similarity representation.
Then, in the inference phase, we average the similarities predicted by the MPSM (dis) and the MPSM (dir) for retrieval evaluation.
EXPERIMENTS 4.1 DATASETS AND IMPLEMENTATION DETAILS
The Flickr30K dataset [Plummer et al., 2015] and MSCOCO dataset [Lin et al., 2014] (1K and 5K test set) were used for validating the effectiveness our proposed method.
We utilize the typical Recall@K (K=1, 5, 10) as the performance evaluation metric.
We trained our model with Adam optimizer with 30 epochs on the Flickr30K dataset andepochs on the MS-COCO dataset.
The dimensionality of the similarity representation to 256, and the other parameters are set to: l = 3 , γ = 0 .
2 , λ = 9 .
4. QUANTITATIVE RESULTS
We compare the proposed MPSM with several state-of-theart baselines.
Note that the majority of these models are ensemble models.
Therefore, we provide two versions of
MPSM: MPSM (dis) and MPSM (dir) that based on the "distance" and the "direction" similarity representation, respectively.
Then, we integrate them by averaging their similarity scores, and denotes as MPSM*.
4.2.1 Results on Flickr30K Dataset
The quantitative results on the Flickr30K dataset are shown in Table 1, and it can be observed that our MPSM model outperforms the state-of-the-art in most assessment measures.
Compared with GSMN, our method outperforms it in all metrics.
Unlike GSMN, our approach propagates and aggregates semantic knowledge, rather than performing imagetext matching directly.
Furthermore, we simulate the interaction of global and local alignments, which obtains more comprehensive cross-modal correlations.
Improvements show that propagating semantic information to learn ﬁne-grained intra-modal correlations and incorporating them into crossmodal similarity learning improves matching performance signiﬁcantly.
Our proposed method outperforms other models that use the same word feature learning method (i.e., bi-GRU).
Compared to CAMERA, our method achieves relative R@1 gains of 2.2 % and 1.2 % for I2T and T2I matching, respectively.
However, our method reduces the relative
R@5 and R@10 to 1.4 % and 1.6 % for T2I matching, respectively.
This could be because CAMERA employs a pre-trained BERT.
BERT learns feature representations of words based on a massive corpus, with powerful language representation and sentence processing capabilities.
When compared to SGRAF, a multi-level alignment learning method that also employs vector-based similarity representation, our method achieves relative R@1 gains of 2.4 % and 3 % for I2T and T2I matching, respectively.
Unlike SGRAF, we model the vector-based similarity representation from two perspectives: distance and direction.
Furthermore, our
SRGR module makes the visual-semantic correspondence more ﬁne-grained.
Moreover, our KGID module provides rich semantic information within modalities.
The advancements demonstrate that learning similarity from multiple perspectives, can help with cross-modal alignment.
It’s worth mentioning that compared with MPSM (dis) and
[Table]
MPSM (dir), MPSM* has increased by 2.7 % and 3.4 % in
I2T retrieval, has increased by 2.8 % and 4.2 % in T2I retrieval relative to R@1, respectively.
This demonstrates that the MPSM(dis) and MPSM(dir) models can complement and enhance each other, allowing for a more comprehensive exploration of the correspondence between modalities.
Furthermore, our single model’s retrieval performance is very competitive, demonstrating the effectiveness of our method.
4.2.2 Results on MSCOCO Dataset
Table 1 shows quantitative results for the larger and more complex dataset MSCOCO (1K and 5K test sets).
Our
MPSM surpasses existing approaches in all metrics in the 1K testset.
Compared with GSMN, our method outperforms it in all metrics.
Compared to SGRAF, on I2T and T2I retrieval, our MPSM improves by 1.3 % and 1.8 % , respectively.
The gain in our method’s performance over R@5 and R@ is not as large as it is for R@1, which could be owing to the presence of more interference sources in a big target set for a particular query.
Our MPSM maintains its superiority in the 5K testset.
Our model outperforms SGRAF by 2.5 % and 1.6 % in I2T and T2I retrieval, respectively.
4.3 ABLATION STUDIES AND ANALYSIS 4.3.1 Impact of Different Network Structures
We compare MPSM (dis) and the integrated model MPSM with four other models (based on "distance" similarity representation).
(1) w/o KGID denotes the removal of the whole
KGID module from the model; (2) w/o V-KGID denotes the absence of the visual KGID module; (3) w/o T-KGID denotes the absence of the textual KGID module; and (4) w/o SRGR denotes the absence of the SRGR module.
As shown in Table 2, both MPSM (dis) and MPSM outperform these four types of models.
Speciﬁcally, when we remove the KGID module, the model performance suffers, which justiﬁes the usage of modality-speciﬁc semantic information to investigate ﬁne-grained semantic correlations and modal representations.
Noting that the performance of w/o
V-KGID and w/o T-KGID is better than that of w/o KGID, owing to the inclusion of semantic information from text or image, which can help with cross-modal correspondence investigation.
The performance of MPSM (dis) is superior to that of w/o SRGR, demonstrating the effectiveness of the SRGR module.
It also demonstrates that in-depth exploration of cross-modal similarity relation can facilitate aggregation and enhance similarity for more accurate matching.
[Table]
4.3.2 Impact of Different KGID Layers
We researched the impact layers of KGID modules on performance, gradually increasing the number of KGID layers from 0 to 4 for training and evaluation.
As can be seen in
Table 3, increasing the KGID improves performance.
The model performs best when the number of KGID layers is increased to 3, demonstrating that iteratively propagating semantic knowledge is effective in boosting performance.
This is because, during knowledge dissemination, the KGID module may integrate nodes’ domain information and build connections with related nodes.
The performance of KGID degrades as the number of layers increases to 4.
This could be due to the fact that as the network grows deeper, the noise level rises in tandem with the number of connected nodes, interfering with the learnt correspondence.
As a result, we ﬁnally set the KGID module to 3 layers.
[Table]
4.3.3 Impact of Different Alignment Strategies
We investigated three alignments: (1) global alignment learning strategy, which implies that only global alignment is used in the model; (2) local alignment learning strategy, implies that only local alignment is used; and (3) multi-level alignment learning strategy, which indicates that global and local alignment are combined.
As shown in Figures 3, the model’s performance decreases dramatically when only the global correspondence is considered, without taking into account the relationship between local and global.
Moreover, when compared to local alignment learning, the multilevel alignment learning strategy achieves superior performance.
It demonstrates that global and local alignments can complement each other’s semantic information to achieve more accurate matching by interacting with "part" and "whole".
4.3.4 Impact of Training Strategies
We design two different training strategies, "joint training" and "independent training with integration”, and compare them.
From Table 4, we can see that the "independent training and integration" strategy achieves superior performance than the "joint training" strategy.
On the one hand, the training with individual learners tends to cause underﬁtting or overﬁtting, resulting in insufﬁcient generalization ability of the joint training strategy.
Instead, we train MPSM (dis) and MPSM (dir) separately, and integrate them by calculating their means to complement each other, resulting in an ensemble modal with superior generalization performance.
On the other hand, the "distance" similarity representation focuses on measuring the magnitude of similarity while ignoring directional differences between images and text; the "direction" similarity representation distinguishes the difference between vectors more from direction than numerical value, and thus fails to quantify the image-text correspondence ﬁnely.
However, they are complementary.
Thus, the ensemble model MPSM is based on "distance" and "direction" similarity representations, which can facilitate the exploration of ﬁne-grained cross-modal correspondences.
[Table]
4.4 QUALITATIVE RESULTS AND ANALYSIS
Furthermore, we show the qualitative results of I2T and T2I retrieval on the Flickr30K in Figure 4.
For the I2T retrieval in (a), we show the top-5 retrieved sentences based on our predicted similarity score ranking.
Our model can retrieve
[Figure]
almost all sentences that match the query image; even the incorrect instances have some similarity.
For example, the "men" region corresponds to the word "men", and the men are wearing uniforms which corresponds to the phrase "men in uniform".
Thus, the semantics between the matched sentence "5" and the query image are almost identical.
This is due to the KGID module, which investigates ﬁne-grained correlations between fragments.
In addition, MPSM considers similarity from multiple perspectives and SRGR module explore more comprehensive similarity and more precise matching.
As for the T2I retrieval, we show the top-3 retrieved images and mark the correct results with green boxes.
The top-1 image is the ground-truth, and all other results are close to the sentence’s semantics.
These results demonstrate our model’s ability to perform ﬁner-grained matching.
CONCLUSIONS
In this paper, we propose a ﬁne-grained matching with
Multi-Perspective Similarity Modeling (MPSM) Network for cross-modal retrieval.
Speciﬁcally, we develop a knowledge graph iterative dissemination module that iteratively propagates semantic knowledge to capture ﬁne-grained intramodal correlations and modal representations.
Then, from multiple perspectives, we learn vector-based similarity representations to adequately learn multi-level correspondences.
Further, we designed a relationship graph reconstruction module that focuses on aggregating and improving the similarity between similar modalities to be able to obtain more accurate matches.
Experiments on both datasets show that our network is superior.
Acknowledgements
This work is supported by National Natural Science Foundation of China (Nos. 61966004, 61866004), Guangxi Natural Science Foundation (No. 2019GXNSFDA245018),
Innovation Project of Guangxi Graduate Education (YCSW2022155), Guangxi “Bagui Scholar” Teams for
Innovation and Research Project, Guangxi Talent Highland Project of Big Data Intelligence and Application, and
Guangxi Collaborative Innovation Center of Multi-source
Information Integration and Intelligent Processing.
(Corresponding author: Zhixin Li.)