  Pan-sharpening aims at fusing a low-resolution (LR) multi-spectral (MS) image
and a high-resolution (HR) panchromatic (PAN) image acquired by a satellite to
generate an HR MS image. Many deep learning based methods have been developed
in the past few years. However, since there are no intended HR MS images as
references for learning, almost all of the existing methods down-sample the MS
and PAN images and regard the original MS images as targets to form a
supervised setting for training. These methods may perform well on the
down-scaled images, however, they generalize poorly to the full-resolution
images. To conquer this problem, we design an unsupervised framework that is
able to learn directly from the full-resolution images without any
preprocessing. The model is built based on a novel generative multi-adversarial
network. We use a two-stream generator to extract the modality-specific
features from the PAN and MS images, respectively, and develop a
dual-discriminator to preserve the spectral and spatial information of the
inputs when performing fusion. Furthermore, a novel loss function is introduced
to facilitate training under the unsupervised setting. Experiments and
comparisons with other state-of-the-art methods on GaoFen-2 and QuickBird
images demonstrate that the proposed method can obtain much better fusion
results on the full-resolution images.
