It is known that the inconsistent distributions and representations of different modalities, such as image and text, cause the heterogeneity gap, which makes it very challenging to correlate heterogeneous data and measure their similarities. Recently, generative adversarial networks (GANs) have been proposed and have shown their strong ability to model data distribution and learn discriminative representation. It has also been shown that adversarial learning can be fully exploited to learn discriminative common representations for bridging the heterogeneity gap. Inspired by this, we aim to effectively correlate large-scale heterogeneous data of different modalities with the power of GANs to model cross-modal joint distribution. In this article, we propose Cross-modal Generative Adversarial Networks (CM-GANs) with the following contributions. First, a cross-modal GAN architecture is proposed to model joint distribution over the data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both compete with each other to promote cross-modal correlation learning. Second, the cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form the generative model. They not only exploit the cross-modal correlation for learning the common representations but also preserve reconstruction information for capturing the semantic consistency within each modality. Third, a cross-modal adversarial training mechanism is proposed, which uses two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make the generated common representations more discriminative by the adversarial training process. In summary, our proposed CM-GAN approach can use GANs to perform cross-modal common representation learning by which the heterogeneous data can be effectively correlated. Extensive experiments are conducted to verify the performance of CM-GANs on cross-modal retrieval compared with 13 state-of-the-art methods on 4 cross-modal datasets. Multimedia data with different modalities-including images, video, text, audio and more-are now mixed together and represent comprehensive knowledge. The research of cognitive science indicates that in the human brain, cognition of outside world is through the fusion of multiple sensory organs [26] . However, there is a heterogeneity gap [55] , which makes it quite difficult for artificial intelligence (AI) to simulate this human cognitive process, because multimedia data with various modalities comprises a huge quantity but has inconsistent distribution and representation. Naturally, cross-modal correlation exists among the heterogeneous data of different modalities to describe specific kinds of statistical dependencies [35] . For example, for the image and textual descriptions coexisting in one web page, they may be intrinsically correlated and share a certain level of semantic consistency. Therefore, it is necessary to automatically exploit and understand this latent correlation across the data of different modalities and further construct metrics on them to measure how they are semantically relevant. For addressing this issue, an intuitive idea is to model the joint distribution over the data of different modalities to learn the common representation, which can form a commonly shared space where heterogeneous data is mapped into. Thus, the similarities between them can be directly computed by adopting common distance metrics. Figure 1 contains illustrations of the above framework. In this way, the heterogeneity gap among the data of different modalities can be reduced so that the heterogeneous data can be correlated more easily to realize various practical applications, such as cross-modal retrieval [33] , in which the data of different modalities can be flexibly retrieved at the same time by a query of any modality. Many methods [2, 24, 59] have been proposed based on this idea. Some learn the common representation by modeling the cross-modal correlation so that the similarities between different modalities can be measured directly. These existing methods can be divided into two major categories according to their different models as follows. The first is in a traditional framework, which attempts to learn mapping matrices for the data of different modalities by optimizing the statistical values to project them into one common space. Canonical correlation analysis (CCA) [39] is one of the representative works, which has many extensions such as those described in [8, 12, 54] . The second kind of methods [6, 25, 31, 32] use the strong learning ability of a deep neural network (DNN) to construct a multilayer network. Most of them minimize correlation learning error across different modalities for common representation learning. Recently, generative adversarial networks (GANs) [9] have been proposed to estimate a generative model by an adversarial training process. The basic model of GANs contains two components: a generative model G and a discriminative model D. The generative model aims to capture data distribution, while the discriminative model attempts to discriminate whether the input sample comes from real data or is generated from G. An adversarial training strategy is adopted to train these two models simultaneously, which makes them compete with each other for mutual promotion to learn better representation of the input data. Inspired by the recent progress of GANs, researchers are attempting to apply GANs into computer-vision areas, such as image synthesis [36], video prediction [7], and object detection [23] . The definition of a generative model can be more generalized. It not only generates the original instances, such as image synthesis [36] , but also generates the representation for each vertex in a graph structure, which aims to learn the embedding for network analysis and prediction [49] . Also, it can generate the representations of documents that are relevant to the give query, which are exploited by the discriminate model to achieve a better estimation of document ranking [50] . While all of them aim to approximate the underlying distribution of original inputs. Thus, the idea of adversarial learning can be fully exploited by discriminating the data of different modalities and to generate the common representations for image and text, which aims to model the cross-modal correlation by approximating the joint distributions over the data of different
However, Peng et al. REF aim to use GANs to learn the common representation and boost the cross-modal correlation learning, which is a completely different goal of our method to learning hash functions without the dependence of paired training samples across modalities.
CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
TOMM
Computer Science