Paper: Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is not innovative enough. I'm not an expert in this field, so I'm not sure about it.

Author response:
**Innovation (W1)**: Our work is the first scalable approach for learning a vision-language model for satellite images which *other reviewers* have appreciated. Moreover, to our knowledge, this is the first method using the idea of an intermediary representation to align two different modalities. We believe that our approach is novel and our results show that we can surpass all the existing baselines in performance.

Reviewer comment (Presentation):
The images in the manuscript are of such low resolution that they are difficult to read.

Author response:
**Image size (W2)**: Thank you for this feedback! We have updated the figures with larger fonts and larger images wherever possible. We have also advised readers to refer to the digital version of the paper, wherever the font remains smaller. If some images are still small due to space constraints, please let us know and we can add larger images in the supplementary.

Reviewer comment (Novelty):
In Section 1, the authors claim existing remote sensing image-text datasets have only 10k samples. However, as far as I know, the volume of RS5M has reached 5 million.

Author response:
**RS5M (W1)**:  Thanks for bringing this to our attention — we have added RS5M to the related work. For your information, the RS5M dataset contains a subset of images that automatically filter from publicly available text-image datasets. As a result, the images come from a diverse range of sources and are not uniform. For example, even in the examples shown in the RS5M paper, we can see many images taken from a plane window. Additionally, our approach allows more control of the data collection process. For instance, we could collect sentinel-2 satellite data from spectral bands other than RGB, allowing us to construct a VLM that could be used on different spectral bands. RS5M, on the other hand, consists of mostly RGB images and thus could only be used for constructing a VLM for RGB images. In addition, the results released by the RS5M paper show that our model performs better on the EuroSat zero-shot classification task (**63.76%** vs 61.48%).

Reviewer comment (Evaluation):
The authors claim that they developed a VLM without textual annotations. Alignment is a good idea (and probably a cheap solution) to obtain a remote sensing VLM by leveraging existing ones from other domains. However, experiments show that the proposed model is still served as a visual encoder in existing VLM models. GRAFT cannot independently finish vision-language tasks, especially for text generation (VQA, captioning, etc.). Indeed, since it is only aligned with the CLIP visual encoder, its cross-modality ability is limited by the pre-trained CLIP text encoder. A discussion of its limitations being applied to other VLMs should be provided.

Author response:
**Limited Text Generation Capabilities (W3)**: Yes we agree with the reviewer that the text generation capabilities of our model are a limitation. We used the term VLM to describe models such as CLIP and ALIGN[1] in our paper. We haved added this as a limitation of our work and this is certainly an interesting avenue to explore in the future. Additionally, as our results show, we can combine our model with other frameworks– for example with ViperGPT– to solve other tasks such as VQA. We can also combine our VLM with other frameworks such as ClipCap [2] to get more powerful VLMs with more capabilities such as captioning.

Reviewer comment (Reproducibility):
Unfortunately neither the code not the dataset are available for review and the soundness of the paper cannot be well estimated.

Author response:
**Code and Dataset (W1)**: We promise to release the code and dataset upon acceptance.

Reviewer comment (Experiments):
One significant disadvantage of the paper is the fact that it is trained only on the RGB bands of the satellite imagery, significantly reducing applications in agriculture, earth and biological sciences.

Author response:
**Multispectral dataset (W2)**: The main contribution of this paper was to demonstrate that we can train large-scale vision language models for satellite images without direct text annotations, by using ground images as an intermediary. Nothing stops us from training a model with multispectral images. To show this, we train another VIT-B/16 model on 100k multispectral sentinel images-ground image pairs (To account for the change in input channels, we change the patch embedding input dimension from 3 to 12. The weights for the RGB channels are initialized to CLIP’s weights, and the rest are set to zero). On the EuroSat multispectral dataset, this model leads to a classification accuracy of 53.78, which is an improvement of **1.26%** on the RGB model trained with the same number of images. We have added this result to the supplementary.

Reviewer comment (Experiments):
It's unclear whether there is a connection b/w ground and satellite imagery by timestamp (to monitor things in "almost real time" or to account for seasonality).

Author response:
**Temporal alignment (Q1)**: The temporal revisit for sentinel-2 is about a week. For every Flickr image, we collect the temporally closest cloudless satellite image (see page 6). The Flickr images in our dataset go back to 2014 while the earliest sentinel images we can get are from 2017. For the first three years of Flickr data, we sample the closest seasonally aligned images. On average the temporal gap between a Flickr image and a sentinel image is around 21 days. So while it is difficult to connect ground images with satellite images of the same day, we can still align seasonality. We have provided this information in the revision.

Reviewer comment (Evaluation):
Paragraph 3.2: you mention accurate geo-location. How do you measure/verify that the geolocation is accurate?

Author response:
**Errors in spatial locations (Q2, 3)**: The Flickr API also provides accuracy in the geotags along with the geotags. We only use the images with the highest level of geotag accuracy. However, it is not unreasonable to think that some of these geotags can still be incorrect. Thankfully, prior work on image-text contrastive learning such as OpenCLIP has shown that the training loss and architectures for such vision language models are robust even at a lower signal-to-noise ratio. Secondly, the patches we contrast image features with are big enough (16m (in length) for NAIP and 160m for Sentinel-2) that our model is invariant to a small amount of noise in geotagging.

Reviewer comment (Evaluation):
How do you account for typical errors in geotagging (~8m)?

Author response:
**Errors in spatial locations (Q2, 3)**: The Flickr API also provides accuracy in the geotags along with the geotags. We only use the images with the highest level of geotag accuracy. However, it is not unreasonable to think that some of these geotags can still be incorrect. Thankfully, prior work on image-text contrastive learning such as OpenCLIP has shown that the training loss and architectures for such vision language models are robust even at a lower signal-to-noise ratio. Secondly, the patches we contrast image features with are big enough (16m (in length) for NAIP and 160m for Sentinel-2) that our model is invariant to a small amount of noise in geotagging.

Reviewer comment (Experiments):
Section 3.4: do you check how old are the features on open street maps?

Author response:
**Open street maps (Q4)**: In section 3.4, we collect the latest NAIP images (2021 and 2022). We also collect the OSM data from the same year instead of the latest one. We have made this information clear in the revision.