  Accurately matching visual and textual data in cross-modal retrieval has been
widely studied in the multimedia community. To address these challenges posited
by the heterogeneity gap and the semantic gap, we propose integrating Shannon
information theory and adversarial learning. In terms of the heterogeneity gap,
we integrate modality classification and information entropy maximization
adversarially. For this purpose, a modality classifier (as a discriminator) is
built to distinguish the text and image modalities according to their different
statistical properties. This discriminator uses its output probabilities to
compute Shannon information entropy, which measures the uncertainty of the
modality classification it performs. Moreover, feature encoders (as a
generator) project uni-modal features into a commonly shared space and attempt
to fool the discriminator by maximizing its output information entropy. Thus,
maximizing information entropy gradually reduces the distribution discrepancy
of cross-modal features, thereby achieving a domain confusion state where the
discriminator cannot classify two modalities confidently. To reduce the
semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss
are used to associate the intra- and inter-modality similarity between features
in the shared space. Furthermore, a regularization term based on KL-divergence
with temperature scaling is used to calibrate the biased label classifier
caused by the data imbalance issue. Extensive experiments with four deep models
on four benchmarks are conducted to demonstrate the effectiveness of the
proposed approach.
