  With the development of earth observation technology, massive amounts of
remote sensing (RS) images are acquired. To find useful information from these
images, cross-modal RS image-voice retrieval provides a new insight. This paper
aims to study the task of RS image-voice retrieval so as to search effective
information from massive amounts of RS data. Existing methods for RS
image-voice retrieval rely primarily on the pairwise relationship to narrow the
heterogeneous semantic gap between images and voices. However, apart from the
pairwise relationship included in the datasets, the intra-modality and
non-paired inter-modality relationships should also be taken into account
simultaneously, since the semantic consistency among non-paired representations
plays an important role in the RS image-voice retrieval task. Inspired by this,
a semantics-consistent representation learning (SCRL) method is proposed for RS
image-voice retrieval. The main novelty is that the proposed method takes the
pairwise, intra-modality, and non-paired inter-modality relationships into
account simultaneously, thereby improving the semantic consistency of the
learned representations for the RS image-voice retrieval. The proposed SCRL
method consists of two main steps: 1) semantics encoding and 2)
semantics-consistent representation learning. Firstly, an image encoding
network is adopted to extract high-level image features with a transfer
learning strategy, and a voice encoding network with dilated convolution is
devised to obtain high-level voice features. Secondly, a consistent
representation space is conducted by modeling the three kinds of relationships
to narrow the heterogeneous semantic gap and learn semantics-consistent
representations across two modalities. Extensive experimental results on three
challenging RS image-voice datasets show the effectiveness of the proposed
method.
