  Retaining spatial characteristics of panchromatic image and spectral
information of multispectral bands is a critical issue in pansharpening. This
paper proposes a pyramid based deep fusion framework that preserves spectral
and spatial characteristics at different scales. The spectral information is
preserved by passing the corresponding low resolution multispectral image as
residual component of the network at each scale. The spatial information is
preserved by training the network at each scale with the high frequencies of
panchromatic image alongside the corresponding low resolution multispectral
image. The parameters of different networks are shared across the pyramid in
order to add spatial details consistently across scales. The parameters are
also shared across fusion layers within a network at a specific scale.
Experiments suggest that the proposed architecture outperforms state of the art
pansharpening models. The proposed model, code and dataset is publicly
available at https://github.com/sohaibali01/deep_pyramid_fusion.
