Paper: DiffusionSat: A Generative Foundation Model for Satellite Imagery (ICLR-2024)

Reviewer comment (Novelty):
Limited technical novelty. The proposed method reduces to a straightforward way of embedding metadata and then conditioning a stable diffusion model on said metadata. The temporal transformer component is from VideoLDM.

Author response:
**Q: Technical Novelty**  
We would like to stress that part of the novelty lies in identifying the usefulness of metadata towards controlling useful generations from latent diffusion models. Natural images simply use semantic text captions to generate images since these are readily available. We recognize the importance of remote-sensing specific numerical metadata towards generating satellite images that are aligned with season, weather, location, resolution etc. This positively affects not only generation quality, but also allows users to prompt the model in more flexible and fine-grained manner (i.e. with/without detailed text, specific months, resolutions, etc.)

While the temporal transformer is introduced in VideoLDMs, our method uses them as part of a conditioning model. Our temporal controlnet is slightly different from the VideoLDM temporal layers; the 3D convolutions of our controlnet operate by default(i.e. they don’t have the learned mix-in parameter), but the temporal attention layers are mixed-in with a learned parameter. The reason is that we swap out 2D convolutions from the controlnet, and so we require some temporal component to always be active (that is, the 3D convolutions). We did experiment with using the temporal transformer by default, but this didn’t work as well. 

We would also argue that using a 3D controlnet in this way is novel, as it unlocks the capabilities of a single-image pretrained LDM (eg: DiffusionSat) to generate arbitrarily long sequences of images auto-regressively. We can do this by either conditioning on a seed sequence of images (eg: real satellite images of a location) and then generate frames in the past, future, or in-between, or generate a completely synthetic sequence of images by sampling the first image of the sequence from single-image DiffusionSat and then using the synthetic image as the seed. In fact, we demonstrate this capability in the appendix, in figure 8, in the revision.

The combined single-image, 3D controlnet approach using numerical metadata allows DiffusionSat to be very general. Control signals of almost any format can be input to our model, whether by numerical metadata or image sequences, to generate a desired output satellite image. This approach to generative modeling for remote sensing data is novel and very flexible.

Reviewer comment (Evaluation):
Quantitative performance across multiple tasks is strong, but not convincingly better than existing approaches. While the proposed method does achieve SOTA performance on some metrics on some benchmarks, it doesn't show consistent enough performance across the board to substantiate the claim that the method is SOTA on all these tasks, as the introduction indicates.

Author response:
**Q: Quantitative results performance**  
Thank you for your observation. We would like to emphasise that some of the quantitative metrics don’t reveal the full story. Metrics such as SSIM/MSE/PSNR can be optimized by samples that aren’t perceptually realistic [1,2], which can indeed be seen by our generated samples we have included in figures 4 and 5. DBPN doesn’t produce many sharp details or any realistic structures, but is able to achieve a high PSNR/MSE. Our samples demonstrate DiffusionSat’s strong ability to super-resolve on the order of 10-20x and predict the past and future changes to images, including on buildings/construction, crop maturity, season, and location. Given these advancements, we claim that DiffusionSat is state-of-the-art in many generative tasks for remote-sensing.

Reviewer comment (Novelty):
The MSI super-resolution problem aims to synthesize 3-band WV3 imagery from 13-band Sentinel-2 imagery. While this may be useful in some scenarios, it does leave out a fairly significant problem, namely, generating all spectral bands. As such, the method does not really perform MSI synthesis, but rather can support MSI inputs for RGB super-resolution.

Author response:
**Q: Multi-spectral super-resolution task definition**  
Thank you for the observation. We should have been clearer. To clarify, the super-resolution task in the paper is one of super-resolving multi-spectral input images to their high-resolution RGB counterparts. With a suitable dataset, DiffusionSat would also be able to generate super-resolved spectral bands given low-resolution conditioning signals, which we leave for future work.

Reviewer comment (Presentation):
Figure 4 is not informative as the disaster-related damage is difficult to identify. Consider highlighting damaged areas, or use a more clear example.

Author response:
**Q: Inpainting figure visibility**  
Thank you for the suggestion. We have included a new figure with more detailed generated samples for the in-painting section. Our qualitative results for in-painting demonstrate DiffusionSat’s strong capability to in-paint damaged regions across a variety of disasters. We also demonstrate the ability to introduce damage to images, based on the type of natural disaster. We hope that this tool is useful to researchers and first-responders who wish to understand damaged or obscured structures in a region in the event of a disaster (eg: which roads are flooded/blocked, which houses were razed, how the landscape changes due to disaster etc.)

Reviewer comment (Writing):
Minor editing problems: leftover question mark (page 4), repetitive text (bottom of pages 1 and 3), typo and highlighting on page 8, etc.

Author response:
**Q: Formatting**  
We have included these changes, thank you for catching them!

Reviewer comment (Experiments):
For the fMoW super-resolution task, it is unclear if the super-resolved images are multi-spectral or just RGB. The text seems to suggest that the model takes in a 13 band Sentinel-2 image and outputs a reconstruction of a 3 band WorldView-3 image from fMoW. If the reconstruction is limited to just 3 bands (R, G, B), then the task is not actually multi-spectral super-resolution.

Author response:
**Q: Multi-spectral super-resolution task definition**  
Thank you for the observation. We should have been clearer. To clarify, the super-resolution task in the paper is one of super-resolving multi-spectral input images to their high-resolution RGB counterparts. With a suitable dataset, DiffusionSat would also be able to generate super-resolved spectral bands given low-resolution conditioning signals, which we leave for future work.

Reviewer comment (Experiments):
**Results for certain tasks are missing or incomplete.** The paper mentions that they show state-of-the-art results for super-resolution, temporal generation, and in-painting. However, only a single qualitative example is provided as result. Also multiple other relevant approaches have been proposed. Superresolution results just compare the proposed approach with Stable Diffusion baseline but ignores the line of work done in the field including [1,2] and others.

Author response:
**Q: Missing results**  
Thank you for your observation. We have added results comparing DiffusionSat with other baselines to the super-resolution and temporal generation tasks. We demonstrate an improvement over prior state-of-the-art super-resolution models including StableDiffusion [1], DBPN [2], and Pix2Pix [3]. We also include qualitative samples for the super-resolution task. These results demonstrate DiffusionSat’s strong capability to super-resolve very low resolution input images to reveal broad structures and landscape. Other methods typically struggle with very low resolution inputs, as can be seen with DBPN producing blurry images. StableDiffusion produces realistic-looking images, but more frequently hallucinates details.

For temporal generation, we include Masked Conditional Video Generation (MCVD) a state-of-the-art conditional video generation diffusion model. Our results demonstrate an improvement over this method, as well as over StableDiffusion adapted with out 3D controlnet. Qualitative samples reflect our method’s improvement over prior works, with more realistic images generated both in the past and the future.

We hope that these results further strengthen our case for DiffusionSat and its role as a generative foundation model for remote sensing data.

Thank you also for suggesting other super-resolution baselines. We have cited these works as part of our related work section. However, were unable to locate public code repositories in order to reproduce their method for our paper. We are happy to include the method’s results given pointers to their implementation. We have included other methods with publicly available code to ensure fair comparison with our method and avoid potential bugs from re-implementing previous works from scratch.

Reviewer comment (Presentation):
**The paper write up needs work to improve cohesion.** The related work section could be merged into the background. The task description coud be joint with the results since bothe are very brief.

Author response:
**Q: Writing cohesion**  
Thank you for the suggestion. We hope to ease the reader into relevant diffusion-model terminology via the background section, which is relevant as we describe our method in the Method sections. We provide a summary of related work towards the end of the paper to aid the reader between the methods and experiments section for better cohesion. 

We have updated our method section to move the description of the datasets used for control signal conditional generation to the experiments section. We have also added further detail in the appendix, including a preliminary investigation into sample quality bias across geography and image-unconditional sequence generation. We hope that these additions improve cohesion.

Reviewer comment (Experiments):
**Overall evaluation and results comparison.** There are multiple other approaches for satellite image generation including conditional that this approach does not compare against. [1] If the focus wants to be on using this model as foundational model for other geospatial ml tasks, then comparison with other self-supervised and/or foundational models is important.

Author response:
**Q: Other baseline comparison results and more examples of generated images**  
Thank you for the suggestion. We have also included more baseline comparison results as mentioned above and in the overall response. In addition, we have provided samples for the temporal generation and super-resolution tasks in the paper, comparing the quality of our results with prior works. 

In the appendix, we also include a brief section on geographical bias, and samples from our model for unconditional temporal generation (i.e. without conditioning on an input image). We hope that these samples demonstrate the quality of DiffusionSat.

Reviewer comment (Presentation):
It is very difficult to see the inpainted damage referred to in Figure 4. Please consider highlighting the area where damage has been inpainted.

Author response:
**Q:  Inpainting damage figure limited visibility**  
We have updated this section to show more samples for the in-painting task. Our in-painting section now properly describes our qualitative results on this task. Since the task is very similar to temporal prediction, we defer the reader to refer to those quantitative results for comparison with prior works.

Our qualitative results for in-painting demonstrate DiffusionSat’s strong capability to in-paint damaged regions across a variety of disasters. We also demonstrate the ability to introduce damage to images, based on the type of natural disaster. We hope that this tool is useful to researchers and first-responders who wish to understand damaged or obscured structures in a region in the event of a disaster (eg: which roads are flooded/blocked, which houses were razed, how the landscape changes due to disaster etc.)

Reviewer comment (Writing):
Abstract missing dot after "remote sensing datasets".

Author response:
**Q: Typos and formatting**  
Thanks for the suggestions, we have taken a closer look and fixed some formatting and typos.

Reviewer comment (Experiments):
Share more examples of generated images.

Author response:
**Q: Other baseline comparison results and more examples of generated images**  
Thank you for the suggestion. We have also included more baseline comparison results as mentioned above and in the overall response. In addition, we have provided samples for the temporal generation and super-resolution tasks in the paper, comparing the quality of our results with prior works. 

In the appendix, we also include a brief section on geographical bias, and samples from our model for unconditional temporal generation (i.e. without conditioning on an input image). We hope that these samples demonstrate the quality of DiffusionSat.

Reviewer comment (Novelty):
The necessity and motivation of designing the generative foundation model are not clear and convincing.

Author response:
**Q: Motivation for generative foundation models + “What is the purpose of conducting research on the generative foundation model?**  
For natural images, generative foundation models used as priors have led to major improvements in a variety of inverse problems like inpainting, colorization, deblurring [1,2,3], medical image reconstruction [4]. Just as StableDiffusion uses a different architecture and training objective to generate images compared to ViTs that are pre-trained for natural image discriminative tasks (eg: MAE [5], MoCo [6], DINO [7] etc.), we are proposing a different architecture and training objective for remote sensing generative tasks.

Remote sensing data provides unique inverse problems that are different from those in natural images. For example, super-resolving widely available multi-spectral Sentinel-2 enables object detection and classification for regions of the world which lack high-resolution satellite data [8]. High-resolution satellite images are also crucial for crop-phenotyping [9], for conservation efforts in biology [10, 14], and landslide hazard assessment [11], 3D city modelling [12], archaeological prospection [13]. Other inverse problems such as cloud/shadow removal have important uses in urban planning [15,16,17]. Further, temporal generation and multi-spectral imputation have benefits towards precise agricultural applications [18].

These studies demonstrate the real-world importance of solving generative tasks using remote-sensing data for industrial and academic use-cases. However, the challenge is that these papers introduce solutions tailored to solve their specific task, and these solutions cannot generalize to multiple downstream generative tasks in one unified framework. We therefore require more research from the generative perspective for remote sensing and our work is a significant step towards this goal.

Reviewer comment (Novelty):
The proposed foundation model for satellite images is not novel and significant since there are many foundation models in the remote sensing area, such as [1-2].  
[1] A Billion-scale Foundation Model for Remote Sensing Images. arXiv:2304.05215.  
[2] Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model. TGRS, 2022.

Author response:
**Q: The proposed foundation model for satellite images is not novel and significant since there are many foundation models in the remote sensing area**  
We believe the reviewer has misunderstood the context of our work and has made a hasty judgement brushing off DiffusionSat’s novelty by dismissing the importance of generative models. Neither of the methods the reviewer cited [19, 20] are generative, which further demonstrates the need for such large-scale generative models in this field and novelty of our work. 

Claiming that our model is “not novel since there are many foundation models in the remote sensing area” is a rather simplistic view that groups all large-scale models as “foundation models” without noting fundamentally different and important  distinguishing factors, i.e. generative model vs discriminative models. This is unfair to the author, other reviewers, and ACs who will take these hasty judgements into account.

To go into more detail, [19,20] are both large-scale, pretrained vision transformers which can solve downstream discriminative tasks, just like [21] which we mention in our paper as a discriminative foundation model. To ask why they cannot solve generative tasks is akin to asking why a self-supervised method for discriminative tasks such as MoCo [6] cannot generate images from captions like StableDiffusion can. They have fundamentally different theory and design/architecture.

Reviewer comment (Novelty):
The authors claim that "while foundation models have been recently developed for discriminative learning on satellite images, no such foundation model exists for generative tasks." Please explain why the existing foundation models are not applicable to the generative tasks.

Author response:
**Q: “Please explain why the existing foundation models are not applicable to the generative tasks.”**  
As mentioned above, the reviewer seems to conflate generative and discriminative models. Discriminative models are not designed to and thus cannot solve any of the inverse problems we mention, including super-resolution, cloud/shadow removal, temporal interpolation.

Reviewer comment (Novelty):
The methodology of training the proposed foundation model is not novel and the authors only apply the stable diffusion model to the satellite images. Although the authors use some new conditional inputs in the stable diffusion framework, it is not novel enough for publication in this conference.

Author response:
**Q: “The methodology of training the proposed foundation model is not novel and the authors only apply the stable diffusion model to the satellite images”**  
We want to stress that our work is more than simply “applying stable diffusion to satellite images”. We demonstrate the effectiveness of simply fine-tuning and applying vanilla stable diffusion to satellite images in all of our experiments: in tables 1, 2, 3, 4. All those results demonstrate that StableDiffusion by itself is insufficient to successfully perform generative tasks with satellite data. For single-image generation, SD is unable to capture variation in season, location, or resolution. For super-resolution, it often “hallucinates” results. Similarly, its performance is worse than DiffusionSat for both temporal generation and texas-housing super-resolution.

We clearly demonstrate that applying stable diffusion to satellite images is non-trivial. Given the differences in satellite images with natural data that we mention in the paper (i.e. irregular time intervals, resolutions, multi-spectral bands, correspondence with “global metadata” etc.), we demonstrate the need for a more thoughtful approach towards designing a generative foundation model for satellite images. 

DiffusionSat does this; our novelty is in recognizing the important ingredients towards designing the first diffusion-based generative foundation model for remote sensing data. We demonstrate the importance of metadata and associating images with numerical metadata in addition, and even in the absence of, semantic text captions. We design a very general 3D conditioning mechanism that enables DiffusionSat to generate images conditioned on a variety of control signals, including images at different time-steps, resolutions, and multi-spectral bands. Further, we demonstrate, qualitatively and quantitatively, the improvements of our method on a collection of generative tasks. The reviewer’s comments about our methodology “only applying stable diffusion to satellite images” or that it is simply “a combination of StableDiffusion and contolnet” come across as disingenuous.

We hope that our work’s novel approach spurs interest and further activity in an underexplored, but highly important field: solving generative tasks for remote sensing data via foundational models.

Reviewer comment (Experiments):
**(W1)**: the presented work is very domain specific i.e., remote sensing data. It would be interesting to see if this approach is able to generalize to other datasets of similar multi-spectral data.

Author response:
**Q: Domain specificity of DiffusionSat, exploring its generality beyond remote sensing?**  
Thank you for your suggestion. Our goal is to demonstrate the importance of DiffusionSat as a foundation model tailored to tackle the unique generative tasks provided by remote-sensing datasets. For example, performing super-resolution on widely available multi-spectral Sentinel-2 data can enable object detection and classification for regions of the world which lack high-resolution satellite data [1]. Other studies demonstrate the benefits of high-resolution satellite data for crop-phenotyping [2], for conservation efforts in biology [3], and landslide hazard assessment [4]. Further studies have shown the importance of cloud or shadow removal in satellite images of urban areas [5] towards urban planning.

These references demonstrate the slew of important inverse-problems associated with remote-sensing data which have broad societal implications. They also demonstrate the growing field of using machine-learning techniques for remote-sensing data. However, each of the works above tailors a specific solution to solve the generative task at hand, and therefore that solution often only applies for their use-case. The important advantage and contribution through DiffusionSat is its potential to tackle multiple of these inverse problems at once. We provide researchers and practitioners an easy way to use DiffusionSat to solve a generative task involving remote-sensing data. 

Thus, we intentionally evaluate and demonstrate the usefulness of our solution on remote sensing data, with the aim of providing a general-purpose solution and with the hope to spur future interest and research in the field of generative foundation models for this data modality. 

We agree that extending and applying DiffusionSat on other forms of multispectral data would be a valuable future line of research inquiry. While we aren’t aware of any such large-scale dataset, we would be happy to accept pointers for the same for future work.

Reviewer comment (Experiments):
**(W2)**: I am confused by the "4. Experiment" section. It is not always straightforward to link the tables and images to the different generative downstream tasks presented. And it seems that some results are missing (i.e., the In-painting section/paragraph is missing entirely)? This point also goes hand in hand with my question Q1, Q2, Q3 below: you mention different downstream tasks in the abstract and the method section, and then the results are not consistently reporting on these downstream tasks (as it seems to me, but maybe I misunderstand something). This is the strongest weakness as it renders the evaluation of the methods to be very difficult for the reader of the submitted paper.

Author response:
**Q: Clarification on the number/types of downstream tasks**  
Thank you for your observation. We apologise for the structure. We want to clarify that our downstream tasks are: (1) multi-spectral input super-resolution, (2) temporal generation/prediction (3) temporal in-painting for natural disaster datasets of satellite imagery. 

The single-image generation tasks (which includes generating single-images given a text prompt and metadata information) are for pre-training DiffusionSat and are not downstream tasks. We first train DiffusionSat on single-image generation, and then use this model for downstream conditional generation on the 3 tasks mentioned above. 

**Q: Incomplete in-painting section**  
Thank you for the observation. We apologise; we meant to include a description of this section that was unintentionally omitted in the first submitted version. We have included a description of the task along with the generated samples with a clearer description to aid readability. We describe our changes in the overall response. We hope this adds further clarity.

Reviewer comment (Writing):
**(Q1)**: Question for clarification, in your abstract you have mentioned 3 downstream tasks (temporal image generation, multispectral image super resolution, and image in-painting. In your methods section you have mentioned 5 downstream tasks (single-image generation, conditioned on text and metadata, multi-spectral superresolution, temporal prediction, and temporal inpainting). Do you summarize those 5 into the 3 in the abstract? And if yes, why?

Author response:
**Q: Clarification on the number/types of downstream tasks**  
Thank you for your observation. We apologise for the structure. We want to clarify that our downstream tasks are: (1) multi-spectral input super-resolution, (2) temporal generation/prediction (3) temporal in-painting for natural disaster datasets of satellite imagery. 

The single-image generation tasks (which includes generating single-images given a text prompt and metadata information) are for pre-training DiffusionSat and are not downstream tasks. We first train DiffusionSat on single-image generation, and then use this model for downstream conditional generation on the 3 tasks mentioned above.

Reviewer comment (Presentation):
**(Q2)**: The "Experiment" section presents results for the following downstream tasks: "4.1 Single Image Generation", "4.2 Conditional Generation". Then in "4.2 Conditional Generation" results are shown for "super resolution" and "temporal prediction". Finally, the "in-painting" section is missing. This is not in sync with the previously mentioned downstream tasks. Could you clarify the structure of the experiment section?

Author response:
**Q: Clarification on the number/types of downstream tasks**  
Thank you for your observation. We apologise for the structure. We want to clarify that our downstream tasks are: (1) multi-spectral input super-resolution, (2) temporal generation/prediction (3) temporal in-painting for natural disaster datasets of satellite imagery. 

The single-image generation tasks (which includes generating single-images given a text prompt and metadata information) are for pre-training DiffusionSat and are not downstream tasks. We first train DiffusionSat on single-image generation, and then use this model for downstream conditional generation on the 3 tasks mentioned above. 

**Q: Incomplete in-painting section**  
Thank you for the observation. We apologise; we meant to include a description of this section that was unintentionally omitted in the first submitted version. We have included a description of the task along with the generated samples with a clearer description to aid readability. We describe our changes in the overall response. We hope this adds further clarity.

Reviewer comment (Experiments):
**(Q3)**: In Table 2, results are presented and compared to other generative methods. Why do you not provide this comparison for Table 1 and Table 3, where results are only compared against non-adapted stable diffusion? Is there any reason for this, if yes, it might be missing in the text body of section 4. I would also recommend to cite the methods you compare to in Table 2.

Author response:
**Q: Other baselines for superresolution and conditional image generation tasks**  
Thank you for the suggestion. Table 1 is intended to demonstrate the effectiveness of our design choices for pre-training the single-image version of DiffusionSat. We compare with StableDiffusion as the largest, most effective single text-to-image publicly available as of the time of writing (which beats most other generative baselines)[6]. Our results in-turn demonstrate the effectiveness of conditioning on text and metadata for DiffusionSat. The methods in table 2 are task specific and unlike DiffusionSat, they cannot be applied for other relevant generative tasks. For example, STSR or Pix2Pix cannot perform single-image generation given text or metadata. 

We have updated Table 3 (now table 2) to include further baselines for super-resolution, based on related work. As can be seen, DiffusionSat outperforms these prior works on the super-resolution tasks. We want to emphasize that the quantitative metrics don’t reveal the full-story- metrics such as SSIM/PSNR/MSE can be high for baselines that produce very fuzzy or unrealistic samples (eg: DBPN [7]). LIPIPS better reflects visual “realism” and our qualitative results also demonstrate the high quality of our generated samples. The drawbacks of SSIM/PSNR are a known problem [8,9].  

**Q: “why are results are only compared against non-adapted stable diffusion”**  
As mentioned above, we do include additional baselines for comparison. We wanted to emphasize the strength of the StableDiffusion baseline, which has been shown across a variety of natural image generative tasks to outperform other baselines[6], including the ones cited in other tables in our work. Therefore, being able to outperform StableDiffusion by strong margins underscores the benefit of our work: our pre-training on text-metadata specific to satellite imagery unlocks generative models’ potential on the unique properties of satellite images.