Paper: Lightweight, Pre-trained Transformers for Remote Sensing Timeseries (ICLR-2024)

Reviewer comment (Experiments):
Limited Spatial Context: Presto is designed to process pixel-time series data and does not process very high-resolution imagery natively. This limitation may impact its performance on tasks where spatial information is crucial, such as scene classification challenges. Image-based models that can distinguish the shape of relevant pixels may be better suited for such tasks.

Author response:
We agree with your points in Weaknesses 1, 2, and 3 (which are concerned with Presto’s processing of pixel-timeseries as opposed to spatial information). While we highlight these weaknesses in our paper, Presto is still performant on scene classification tasks - in particular, we show the performance vs. parameter / FLOPs tradeoff in Figure 3 of the updated paper, where Presto achieves competitive results with state of the art models for the EuroSat task while requiring orders of magnitude less parameters & FLOPs.

     i. **Limited spatial context**: We agree that for scene classification tasks where this spatial context is important, Presto may perform sub-optimally. We discuss this in more detail in Section A.7. However, for many remote sensing tasks spatial context is much less important than temporal context, and it is therefore common for remote sensing practitioners to train pixel-timeseries models [1,2,3,4,5] (we discuss this in Section 1 of our paper). Presto addresses the use case of these practitioners explicitly, while remaining performant on scene classification tasks.

Reviewer comment (Evaluation):
Plateauing Performance with Increasing Input Resolution: The performance of Presto on the EuroSat dataset reaches a plateau as the input resolution increases. This plateauing is mainly caused by a failure to accurately predict specific classes, such as the Highway and River classes. This suggests that Presto may struggle with accurately classifying images where the relevant pixels for a class are a minority of the pixels in the image.

Author response:
iii. **Plateauing performance with Increasing input resolution**: We agree that Presto may struggle with accurately classifying images where the relevant pixels for a class are a minority of the pixels in the image. We explore this in more detail in Section A.7, including investigating performance for specific classes in Figure 7.

Reviewer comment (Evaluation):
Lack of Discussion on Specific Limitations: While the paper briefly mentions limitations related to spatial context and lossy aggregation of spatial information, it does not provide an in-depth discussion of these limitations. Further exploration and mitigation techniques for these limitations are mentioned as future work. A more detailed discussion of these limitations would provide a clearer understanding of the model's constraints.

Author response:
As we note in our point above, we have dedicated significant parts of the paper to discussing potential limitations (including over a page discussing Presto’s performance on tasks where spatial context is important in Section A.7, “Presto’s failure modes”). We have also included a limitations section in Section 5 in the main body of the paper, where we discuss constraints such as Presto’s inability to process very high resolution imagery.

    We would be happy to expand these discussions; is there anything you would specifically like us to explore in more detail?

Reviewer comment (Experiments):
How does Presto handle missing or unreliable labels in remote sensing datasets? Are there any specific techniques or strategies employed to mitigate the impact of limited labeled data?

Author response:
We pre-train Presto to learn representations of remote sensing data. We show in Table 3 that models with very few parameters can be learned on top of these representations, allowing Presto to be used even when very few labels are available.

    For example, Presto excels at the CropHarvest task (in which as few as 203 labels are available, as shown in Table 2).

Reviewer comment (Experiments):
Can Presto effectively handle tasks that require a high level of spatial context, such as scene classification challenges? If not, are there any plans to address this limitation in future work?

Author response:
Presto is competitive with SoTA models in the EuroSat scene classification task, while requiring orders of magnitude fewer parameters and FLOPs (Figure 3). 

    However, as we discuss in Section A.7, improving Presto’s capability even more in this regime is an area of future work.

Reviewer comment (Experiments):
How does Presto perform when the input resolution increases? Are there any specific challenges or limitations observed in accurately predicting specific classes at higher resolutions?

Author response:
In Section A.7 and Figure 7, we discuss Presto’s failure modes. Specifically, we highlight that in scene classification challenges where the predictions are dependent on the occurrence of objects in subregions of the image, Presto’s performance does not improve as input resolution increases. 

    Example classes in the EuroSat class (which we plot in Figures 7 and 8) include the river and highway classes.

Reviewer comment (Novelty):
The application of Transformers to remote sensing should be discussed in more detail, e.g. [1] for pixel timeseries classification and [2] image timeseries classification and segmentation (despite operating on image timeseries base their model design on similar points as bullet points 1, 2 mentioned in the introduction).  
[1] https://www.sciencedirect.com/science/article/pii/S0924271620301647  
[2] https://openaccess.thecvf.com/content/CVPR2023/html/Tarasiou_ViTs_for_SITS_Vision_Transformers_for_Satellite_Image_Time_Series_CVPR_2023_paper.html

Author response:
- **Additional discussion of related works**: Thank you for highlighting these papers. We already cite [1] in our introduction, in the paragraph titled “A highly informative temporal dimension”. Based on your suggestion, we add a paragraph in our introduction discussing the use of transformers for remote sensing time-series.

Reviewer comment (Experiments):
The importance attributed to small model size is not as significant for the intended use case as it is presented in the paper. While it is always desirable to reduce model size and FLOPS as this reduces both latency, use of computing resources and their environmental footprint, models operating on satellite images will typically run on servers, and there is a limit to the number of inference runs imposed by the size of the area of interest. The revisit time of satellites (5 days for S2) provides a window for inference to run over large areas even with commodity hardware. This is in contrast to models operating on edge devices or models potentially receiving a very large number of calls on servers, e.g. chatbots, in which small model size can be crucial for deployment. If the authors have some specific use cases under consideration these should be presented and the argument should be made why a small model is needed. Also, one of the main findings of deep learning research, the fact the large overparameterized models outperform smaller ones even for small scale datasets, has not been successfully applied to satellite image timeseries in which performance typically drops after a certain model size. An ablation in terms of model size would be interesting to showcase how the proposed framework scales.

Author response:
- **Importance of model size**: Our apologies for not being clearer in the paper. It is indeed surprising, but in practice, a very large number of remote sensing applications are bottlenecked by computation. We've seen this in our own collaborations with public agencies deploying remote sensing algorithms. Even in rich, well-resourced countries, governments often do not have the computational resources (yet) to deploy highly compute-intensive algorithms, and this is far worse in poorer countries. Most such applications are currently using less effective algorithms to save on computation. We highlight quotes below from practitioners which emphasise the importance of model size - not just for latency, but for total cost:
  - "Becoming Good at AI for Good" [3] by authors from Microsoft, has a subsection titled “Model development with resource constraints”. We quote the following specific use case from this paper:

        “ Robinson et al. [51] trained a fully convolutional neural network (CNN) on over 55 terabytes of aerial imagery to create a high-resolution land cover map over the United States. Differences in seconds of running time per batch translate to hundreds of dollars in the cost of the final computation. Here, a larger, state-of-the-art model would incur a ~270% increase in the cost of the final computation for a fractional increase in performance metrics such as accuracy and intersection-over-union, and so a trade-off in favor of lowering the cost was made.

  - The European Space Agency’s World Cereal program, which mapped cropland at a global scale at a 10m resolution, shared a blog post of their experience (https://blog.vito.be/remotesensing/worldcereal-benchmarking). They emphasise three key criteria when selecting models:

        “(i) Quality-related (e.g. how accurate are the results?), (ii) Performance-related (e.g. what computational resources are needed?), (iii) Requirements-related (e.g. how much and what kind of training data is required?)”

  - The SoilsGrid project [4] mapped multiple soil metrics at a global scale, at a 250m resolution. We quote two areas from their paper where the discuss the computational costs, and modelling considerations as a result of these costs:

        “The total size of all input and output data used to generate SoilGrids exceeds 30 TiB, so that a first step in preparing SoilGrids250m was to obtain a Synology 12-Bay NAS storage server with 60 TiB space. Handling such a large data set presented major challenges considering computational complexity and network bandwidth limitations.”

      They omit kriging, which may increase performance, because of the computational costs:

        “... overall kriging of residuals for global land mass does not seem to be necessary nor is it practical to implement for billions of pixels: it would only marginally improve the accuracy of predictions at high computing costs.”

  - MOSAIKS [5] develops a feature-database from satellite imagery specifically with the goal of reducing the computational cost of applying machine learning algorithms to satellite imagery. We quote their introduction:

        “The resource requirements for deploying SIML technologies, however, limit their accessibility and usage. Satellite-based measurements are particularly under-utilized in low-income contexts, where the technical capacity to implement SIML may be low, but where such measurements would likely convey the greatest benefit. For example, government agencies in low-income settings might want to understand local waterway pollution, illegal land uses, or mass migrations. SIML, however, remains largely out of reach to these and other potential users because current approaches require a major resource-intensive enterprise, involving a combination of task-specific domain knowledge, remote sensing and engineering expertise, access to imagery, customization and tuning of sophisticated machine learning architectures, and large computational resources.”

Reviewer comment (Novelty):
While the authors recognise the need for methods particularly designed for remote sensing data, other than the domain specific inputs, the proposed method itself does not account for such characteristics, utilising a pre-existing framework (MAE) and architectures (Transformers) without significant modifications.

Author response:
- **Adapting the method for satellite data**: We highlight the following ways in which our model architecture is adapted to satellite data:
  - **The use of location metadata as inputs**: While location metadata is always available in remote sensing datasets (e.g. EuroSat), it is rarely used by models. Our method leverages this location metadata.
  - **The use of temporal encodings**: We adapt the traditional positional encodings used by transformer models to additionally consider the month which a token comes from. We discuss this in Section 2.2.
  - **Channel aware encodings**: We additionally include a learnable encoding which communicates to the model which sensor a token came from. This critical detail allows us to include multiple sensor inputs into our model; we are the first MAE-based model to support multiple sensors.

  In addition, we note the following modifications to the MAE pre-training approach: 
  - **A masking strategy with downstream tasks in mind**: As described in Section 2.3, we introduce (and ablate) a masking strategy which is aware of the structure of the data. Missing channels is a unique characteristic of remote sensing data which this method is specifically designed to address.
  - **The introduction of categorical and continuous inputs**: We introduce the ability to ingest categorical and continuous inputs at pre-training (and downstream) time. This allows for information-dense inputs (e.g. landcover maps) to be used in pre-training, allowing for semantic density to be introduced during pre-training.

Reviewer comment (Evaluation):
In Table 1 random timesteps appear to have practically the same performance as all transformations combined.

Author response:
- **In Table 1 random timesteps appears to have practically the same performance as all transformations combined.**

    Yes - however, we note that (in addition to performance gains), an advantage of this pre-training method with combined masking is that it reflects regimes in which the model will be used for downstream tasks (e.g. with missing channels, or missing timesteps or combinations of both). We added a clarification in the text in section 2.3. We also renamed the "Random timesteps" masking strategy to just "Timesteps", to avoid confusion. The combined strategy is clearly better than the "Random" one, which is what we meant in the text.

Reviewer comment (Experiments):
Section 3.2 discusses performance in image-based tasks despite the model not containing mechanisms for spatial modelling, essentially operating as a deep fully connected network on single pixel features. It has to be clarified what can be derived from such a comparison. In Table 4 it appears that both baselines outperform Presto. In Table 5 experiments it is shown that Presto outperforms baselines specifically trained on spatial data, which is very surprising given that Presto has no understanding of spatial relations and that the MAE family of models are very similar in terms of their pre-training objective. This most probably can be attributed to the fact that there is a train/test input size discrepancy for the baselines. Given that both models operate out of domain, not much can be derived from this comparison. If the argument is related to the need for reduced algorithmic complexity, then this is not a fair comparison for the baselines, which could be trained at smaller resolutions and be better and less expensive to compute at the same time. Overall, reduced algorithmic complexity does not justify the use of a non-spatial model in these spatial modelling tasks (see Point 2).

Author response:
- **Section 3.2 discusses performance in image based tasks despite the model not containing mechanisms for spatial modelling, it basically operates as a deep fully connected network on single pixel features. It has to be clarified what can be derived from such a comparison.**

    We agree that Presto is primarily designed for pixel-timeseries. Note that because Presto always gets a lat/lon token as input, in addition to other channel tokens (like the RGB one), there are always at least 2 tokens that can attend to each other in self-attention layers.

    Our goals when benchmarking Presto on scene classification tasks are:
    - **Comparing Presto to comparable self-supervised models.**: Since all globally pre-trained self-supervised models for remote sensing ingest image-based datasets, comparing Presto to works like Seasonal Contrastive Learning, SatMAE or ScaleMAE require us to evaluate Presto on image-based datasets. 
      
    - We show that even a pixel-based small, efficient model with a simple spatial aggregation scheme achieves good performance on some scene classification tasks. Rather than outperforming very large models pre-trained specifically for spatial modelling, we offer a new choice that can be optimal for different preferences w.r.t. the performance/efficiency trade-off. 

    - Demonstrating the **wide applicability of Presto to different types of tasks**. Users might be interested in one model that handles both pixel-based and image-based tasks, for ease of use.

  - **In Table 4 it appears that both baselines outperform Presto.**

    Yes - we discuss this in Section A.3.1, and hypothesize that this is in part due to Presto’s (currently) coarse approach to modelling spatial information.  

  - **In Table 5 experiments it is shown that Presto outperforms baselines specifically trained on spatial data which is very surprising given that Presto has no understanding of spatial relations and that the MAE family of models are very similar in terms of their pre-training objective. This most probably can be attributed to the fact that there is a train/test input size discrepancy for the baselines. Given that both models operate out of domain not a lot can be derived from this comparison.**

    We begin by noting that while Presto approaches the performance of the other MAE models, the best performing model given a full-resolution input is the ScaleMAE model, which achieves an accuracy of 0.960. In terms of train/test discrepancy, we use the (widely used) splits provided by [6], as implemented in the TorchGeo library. These additionally reflect the splits available in the ScaleMAE codebase [[link](https://github.com/bair-climate-initiative/scale-mae/tree/main/mae/splits)], suggesting we used the same train/test splits as the other models. 

    If by input size you are referring to the resolutions: we note that the ScaleMAE includes the exact same experiment (Table 11), where they compare ScaleMAE and SatMAE performance on EuroSat for input resolutions 16, 32 and 64. In addition, the ScaleMAE model has been specifically designed to work for different input resolutions; their abstract states: "we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process." Hence, we do not agree that the baselines are operating out of domain, at least not to a greater extent than Presto (which has also been pre-trained just on 10m/pixel resolution data).

Reviewer comment (Evaluation):
Experimental results should include more performance metrics, including pixel-based and class-based metrics. Tables from supplementary material should be moved to the main paper. Several captions need to provide more details to make them complete for understanding (Table 1: class or pixel-based metric; Tables 4, 5: clarify what are top, bottom).

Author response:
- **Experimental results should include more performance metrics including pixel based and class based metrics. I propose to move tables from supplementary material to main paper.**

    Thank you for this feedback - we have been consistent with the metrics reported by (i) the datasets themselves, and (ii) works which benchmark against them. Which specific metrics would you additionally like to see reported? 

  - **Also, several captions need to provide more details to make them complete for understanding (Table 1: class or pixel based metric, Tables 4, 5: what are top, bottom?).**

    Our apologies for the lack of clarity. Table 1 computes pixel-based metrics.

    - In Table 4, the top and bottom represent different inputs (S1 and S2), as denoted by the “Data” column
    - In Table 5, the top row of Presto represents results with RGB inputs, and the bottom with multispectral (MS) inputs.

    We have updated the tables and the task descriptions in Section 3.2 to make this clearer.

    Please let us know if we have misunderstood which top / bottom you were referring to.

Reviewer comment (Reproducibility):
Will the pretraining data be shared as part of the submission to help replication?

Author response:
- **Will the pretraining data be shared as part of the submission to help replication?**

    Yes, we will publish the pre-training data after publication. All scripts used to export it from Earth Engine are currently available in the anonymous repository (https://anonymous.4open.science/r/presto-C4AB).

Reviewer comment (Experiments):
How does a Transformer trained from scratch in downstream tasks compare with the pre-trained model?

Author response:
- **How does a Transformer trained from scratch in downstream tasks compare with the pre-trained model?**

    We answer this question in part 2 of our answer ([link](https://openreview.net/forum?id=Iip7rt9UL3&noteId=3hJMeoUJxM)) above.

Reviewer comment (Experiments):
Regarding image-based tasks, the reasons why a pixel-timeseries model outperforms spatial-only models in Table 5 should be analysed. Specifically:  
a) What is the baseline performance and FLOPS when trained at test resolution, and what is the performance using 224x224 inputs?  
b) Do both baselines and Presto use the same input data or is Presto informed with additional details about location, elevation?

Author response:
- **Regarding Image based tasks, the reasons why a pixel-timeseries model outperfroms spatial-only models in Table 5 should be analysed. a) What is the baseline performance and FLOPS when trained at test resolution? and what is the performance using 224x224 inputs?**

    We want to emphasize that both SatMAE and ScaleMAE use EuroSat as a benchmark in their paper. ScaleMAE uses another even lower resolution dataset, and has an experiment with ScaleMAE and SatMAE results for the downscaled (16, 32) versions of EuroSat. Their model is both trained and tested on 224x224 inputs, both in their paper and in all results that we include in our paper. We cannot for obvious reasons retrain SatMAE and ScaleMAE on different resolutions, and we do not think that that would be within the scope of our paper.

    We also note that [7] specifies that inputs should be resized to the pre-training image size (in this case 224 x 224 for SatMAE and ScaleMAE) for best results.

  - **Do both baselines and Presto use the same input data or is Presto informed with additional details about location, elevation?**

    In the EuroSat experiments, Presto gets the same inputs as the baselines (RGB or Multispectral S2 data), plus the encoded lat/lon coordinates. No elevation, topography or other input channels are used.

Reviewer comment (Experiments):
How does Presto scale with model size, pretrain-data size, and finetune-data size?

Author response:
- **How does Presto scale with model size/pretrain-data/finetune-data size ?**
    
    Thank you for this suggestion. We add an experiment showing how Presto’s performance scales with model size; we find that as model size increases, Presto’s performance increases when measured against the CropHarvest validation set. The results are available in Table 8 and are discussed in Section A.6.

Reviewer comment (Experiments):
**Resolution constraints**. Most downstream tasks Presto is tested on use coarse resolution. It would be nice to test the effect of encoding different resolution imagery during pretraining.

Author response:
**Resolution constraints**: We agree that Presto is only tested at the resolution of freely available satellite imagery (and highlight this as a limitation and area of future work in Sections 5 and A.3.1 of the paper). We note that the EuroSat runs are run at a variety of (decreasing) image resolutions - Presto’s performance remains relatively consistent as image resolution decreases.

Reviewer comment (Evaluation):
**Small performance gains**. The performance improvement for some of the tasks tested compared to simple baselines is limited.

Author response:
**Small performance gains**: We emphasise that in many cases, we are benchmarking against state-of-the-art methods for a task. The main contribution of Presto is not to achieve state-of-the-art results on some benchmarks; it is instead to introduce a computationally efficient model which can be applied to a large range of remote-sensing data formats. We emphasise that no other pre-trained model would be capable of running on all of these benchmarks, let alone achieving competitive results on them.

    In addition, we do note that performance gains are considerable for CropHarvest (Table 3), and for the EuroSat tasks (Figure 3) (especially when considering the compute vs. performance trade offs).

Reviewer comment (Experiments):
**Set of downstream tasks tested**. It is worrisome to me that most self-supervised methods test their proposed approach against a different set of downstream tasks making it hard to actually compare approaches. There are multiple other self-supervised approaches that could be used as comparison.

Author response:
**Set of downstream tasks tested**: Thank you for this feedback. We have added other models to the EuroSat benchmark, to better illustrate the performance of Presto relative to other self-supervised models and model architectures. Specifically, we have added the Seasonal Contrast (SeCo) and GASSL papers as additional benchmarks for Presto.

    We note that other self-supervised methods **cannot ingest** the full set of downstream tasks we test against (for example due to their inability to ingest the full timeseries or all the sensor inputs). This ability to process diverse modalities of data (which we fully describe in Table 2) is one of Presto’s key strengths.

Reviewer comment (Evaluation):
Do you have intuition on why the improvement from Presto on TreeSatAi is much larger on S1 than S2 on Table 4.

Author response:
**Intuition on TreeSat performance**: Our hypothesis is that the information gain from the implicit reconstruction (i.e. from Presto reconstructing S2, ERA5, Dynamic World and topography given S1) is much higher for S1 than from S2.

Reviewer comment (Presentation):
Figure 2 would benefit by explaining which channels are missing and which channels were masked.

Author response:
**Figure 2 caption**: Apologies for the confusion; in our caption, we state that “In this experiment, we mask the Sentinel-2 RGB channels". There are no other channels masked nor missing - we have updated the caption to specify this. Please let us know if there are any additional changes you would like to see to make this clearer.