SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Unsupervised pre-training methods for large vision models have shown to enhanceperformance on downstream supervised tasks.
Developing similar techniques forsatellite imagery presents signiﬁcant opportunities as unlabelled data is plentifuland the inherent temporal and multi-spectral structure provides avenues to furtherimprove existing pre-training strategies.
In this paper, we present SatMAE, apre-training framework for temporal or multi-spectral satellite imagery based on
Masked Autoencoder (MAE).
To leverage temporal information, we include atemporal embedding along with independently masking image patches acrosstime.
In addition, we demonstrate that encoding multi-spectral data as groups ofbands with distinct spectral positional encodings is beneﬁcial.
Our approach yieldsstrong improvements over previous state-of-the-art techniques, both in terms ofsupervised learning performance on benchmark datasets (up to " 7%), and transferlearning performance on downstream remote sensing tasks, including land coverclassiﬁcation (up to " 14%) and semantic segmentation.
Introduction
In recent years, self-supervised learning techniques have quickly become the norm for pre-trainingmodels on large-scale natural image datasets [1, 2, 3, 4, 5, 6, 7, 8] and have demonstrated strongperformance on downstream tasks including image classiﬁcation [3, 4, 9, 10], image segmentation [3,11], representation learning [12, 13, 14], image compression [12, 15] and image reconstruction[1].
Unlike supervised learning approaches, self-supervised learning techniques do not requirehuman labeling making them appealing in settings where unlabeled data are abundant but labeleddata are scarce, such as remote sensing data (e g., satellite imagery).
While several large-scalesatellite image datasets have been carefully curated in the past few years, including Functional Map of the World (fMoW) [16], BigEarthNet [17], xView [18], SpaceNet [19], annotating thesedatasets requires specialized skills and is more expensive than traditional computer vision datasets.
Moreover, automatic analysis of satellite imagery is often needed for tasks with large societal impactsuch as poverty or crop yield prediction [20, 21, 22, 23, 24, 25, 26, 27, 28], where acquiring largeamounts of labeled data through surveys is impossible or prohibitively expensive.
This suggests thatself-supervised learning approaches for satellite imagery could be especially valuable.
However, existing self-supervised learning approaches [1, 2, 3, 4, 5, 6] are mainly designed for naturalimages.
As opposed to natural images such as ImageNet [29], satellite imagery is usually associatedwith meaningful geographical and temporal information, and can consist of multiple spectral bandsrepresenting various sensor readings besides visible light (i.e., RGB channels typical in naturalimages).
Depending on the data source, satellite imagery can also vary signiﬁcantly in resolution.
While self-supervised learning methods for satellite imagery exist [30, 31], these approaches cannotlearn general representations for both temporal and multi-spectral remote sensing data
[Figure]
t e m p o r a l o r s p e c t r a l d a t a
[Figure]
[Figure]
To address this issue, in this paper, we propose SatMAE , a self-supervised learning frameworkbased on masked autoencoders (MAEs) [1] which naturally handles temporal and multi-spectralinput data.
We show that introducing a positional encoding for the temporal/spectral dimension andindependently masking patches across the temporal/spectral dimension beneﬁts pre-training, allowingthe model to learn representations of the data that are more conducive to ﬁne-tuning.
Speciﬁcally, ourcontributions are as follows
We propose a novel method to leverage temporal or multi-spectral information in satellite
imagery to improve self-supervised pre-training with masked autoencoders (see 4).2.
We introduce fMoW-Sentinel, a new Sentinel-2 dataset cross-referenced with fMoW, as abenchmark for training models on multi-spectral satellite imagery (see 5.1).
We demonstrate the effectiveness of pre-training transformers [32] on satellite imagery,achieving signiﬁcant improvement over previous state-of-the-art methods on benchmarkdatasets as well as downstream remote sensing tasks (see 5)
Related Work
ML for SITS Deep learning has been used for many Satellite Image Time Series (SITS) supervisedlearning tasks such as crop-type mapping [28, 27, 33, 34], yield prediction [35, 36], precipitationforcasting [37], land-cover classiﬁcation [38, 39, 40, 26].
These works establish the usefulness oftailoring architectures such as LSTMs, self-attention, and transformers to temporal data.
However,outside of their speciﬁc task, they are often not directly applicable to other remote-sensing datasets.
SSL for Satellite Imagery Self-supervised learning [2, 3, 4, 5, 6] has emerged as a promisingapproach in remote sensing domains.
For instance, [30] and [31] propose incorporating spatiallyaligned images over time for contrastive self-supervised learning.
Despite promising results, thesetwo contrastive learning approaches rely heavily on the quality of positive pairs, which is oftenhard to control.
[41] combines different sensor channels to generate co-located images that serve aspositive pairs.
[42, 43, 44] apply off-the-shelf contrastive learning algorithms to satellite images.
[44]utilizes image inpainting and transformation prediction as additional pretext tasks.
[45] leveragesgeographical knowledge to aid SSL, which, however, can be difﬁcult to obtain as annotations.
Masked Autoencoder MAE [1] is a recent powerful self-supervised learning method.
Instead ofconstructing a contrastive objective, it proposes the pretext task of reconstructing masked patchesof the input, and largely avoids the need for designing speciﬁc data augmentation.
Inspired by
MAE’s state-of-the-art performance on a wide collection of vision benchmarks [1], many follow-upworks extend MAE to different data modalities.
VideoMAE [46] proposes video tube maskingand reconstruction as a pretext task for video analysis.
GMAE [47] adapts MAE to the domain ofgraphs.
MultiMAE [48] takes optional inputs of different modalities and accordingly includes othertraining objectives to facilitate multi-modality learning.
However, these works fail to optimally handletemporal and multi-spectral input.
VideoMAE requires equally-spaced image frames in the temporaldimension, which is not the case for satellite data given the temporal irregularity and discontinuity insampling images of a location.
In this work, we incorporate temporal and spectral information into amasked autoencoder architecture, and propose a novel self-supervised framework for satellite data.
Background
Masked Autoencoder The MAE is an autoencoder with asymmetrical encoding and decodingstages [1].
It operates on images I 2 R C ⇥ H ⇥ W , where H, W are the height and width of theimage, respectively, and C is the number of channels.
The input image I is resized to a sequenceof non-overlapping patches, S 2 R L ⇥ P 2 C , where P is the height and width of the patch, and
L =
( H/P ) · ( W/P ) is the number of patches.
Each patch is passed through a patch embeddingf p : R P 2 C 7!
R D to create a sequence S 0 2 R L ⇥ D of embedded patch “tokens”.
A fraction p m ofthe L tokens are masked and only the remaining (1  p m ) L “visible" patch tokens are fed to theencoder, a Vision Transformer (ViT) [32] with positional embeddings to capture the spatial locationof the patch in the image.
The decoder is a series of transformer blocks that operates on all L tokens(with positional embeddings added), where the p m L encoded visible patches are placed in theiroriginal sequence position among (1  p m )
L masked patches represented by a learnable mask token.
The decoder outputs a reconstructed image ˆ I 2 R C ⇥ H ⇥ W , which is compared to the original imageusing the mean-squared error (MSE) loss, computed per-pixel only on the masked patches [1].
Positional encoding Positional encoding allows transformers to make its learned representationsposition-aware.
In MAE [1] and many transformers [49, 50], the positional encoding is:
[Equation]
Here, k is the position, i is the index of feature dimension in the encoding, d is the number of possiblepositions, and ⌦ is a large constant (normally set to 10000).
In MAE, position is deﬁned as the indexof the patch along the x or y axes.
Therefore, k ranges from 0 to H/P (or W/P ).
The ﬁnal encodingis generated by concatenating the encodings of the x and y coordinates.
Method
In this section, we describe SatMAE with temporal (4.1) and multi-spectral (4.2) satellite images.
4.1 Temporal SatMAE
We now consider input tensors I T 2 R T ⇥ C ⇥
H ⇥ W , where T denotes the number of images in atemporal sequence.
In video data, T frames are usually equally spaced.
However, temporal satelliteimagery rarely has images at regular intervals.
More commonly, several snapshots, or versions, of agiven location are taken at irregular times.
The length and sample frequency of these sequences ofsatellite images vary drastically over years and across different regions.
Naïvely, one could reshape I T to I
T 2 R TC ⇥
H ⇥ W , effectively concatenating the temporal sequenceof images along the spectral (i.e. channel) dimension, and then apply the MAE machinery verbatim.
This method poses a few difﬁculties: (i) the model may be unable to generalise to a temporal orderingdifferent to the one used in pre-training, since it can only understand order through the position ofimages in the stacked-timeseries (ii) the model cannot reason about the length of time separatingtwo consecutive images in a time sequence, which may be variable when images of a location aresampled at irregular intervals (iii) the model loses access to temporal ﬁne-grained information indeeper layers, as its only direct exposure to encode temporal information is through the initial patchembedding f p (iv) the model is not temporally-shift invariant (i.e. the model would need to separatelylearn to detect the same event in two different segments of a temporal sequence).
To address these challenges and to avoid losing temporal information, we resize the temporal sequence
I T to S T 2 R L T ⇥ P T P 2 C , where L T =
L · ( T/P T ) =
( H/P ) · ( W/P ) · ( T/P T ) , P T is the “patch size”in the temporal dimension, and L and P are deﬁned in 3.
Prior works using transformers for videodata suggest using P T = 2 , where each “patch” is a cube of shape 2 ⇥ 16 ⇥ 16 [46, 51, 52].
Since ourdata has much shorter temporal sequence lengths [16], we let P T = 1 such that L T =
L · T .
In orderto operate on inputs of any temporal order, we re-use the same patch embedding f p : R P 2 C 7!
R Dfor each image in the time series, giving us an embedded sequence of tokens S 0 T 2 R L T ⇥ D .
[Figure]
For each embedded token in the L T length sequence, weneed to ensure the model retains information about itsspatial and temporal position.
As shown in many priorworks [30, 31], the timestamp of a satellite image is usefulfor many pre-training or downstream vision tasks.
Wepropose a temporal encoding scheme compatible with themasked autoencoder architecture by treating the temporaldimension similarly to the positional dimensions (see 3).
The timestamp of a satellite image is represented as “yearmonth-day-hour-minute-second”.
Instead of passing theentire numerized timestamp into a feature encoder, wepropose only keeping the useful parts.
Intuitively, the day,minute, and second should be unrelated to the visual appearance of a region.
Thus, including these components inthe temporal encoding may not be beneﬁcial, and can evenbe detrimental.
In contrast, a landscape may evolve overyears due to weather, geology, and human activity.
Themonth reﬂects season and climate, and the hour reﬂectsdaylight and temperature.
Then, the temporal encoding is formulated as:t k,i =
CONCAT
[ Encode ( k year , i ) , Encode ( k month , i ) , Encode ( k
[Equation]
, i )] (2)
And the ﬁnal encoding is generated by concatenating the temporal encoding to the positional encodingdeﬁned in 3 such that the total length of the encoding is D .
4.1.2 Masking Strategies
With an additional temporal dimension, masking a subset of the L T tokens needs to be treated withcare.
As seen in ﬁgure 3, there are different ways to mask a temporal series of satellite images.
Consistent Masking Each image is “patchiﬁed” separately, but the masked regions are consistentacross all images (ﬁg. 3a).
This approach is also used in VideoMAE [46], with video input.
Independent Masking
Each image is “patchiﬁed” separately, and masked regions may not be thesame across every image.
Instead, a fraction p m of the full sequence of all patch tokens are masked.
Another variant is to independently mask the regions of each image, but keep the ratio p m of maskedregions ﬁxed per image.
Both variants are equivalent in expectation.
Effectively, the model may lookat unmasked values of a region that is masked in one image but not in others.
While this setting maylead to an easier task for video data since the model can “cheat” and exploit temporal redundancy invideos with high framerates [46], we argue that this form of “cheating” is less feasible in temporalsatellite imagery, given the strong impact of seasonal variation and changing human activity overperiods of time and the much larger time deltas between temporally consecutive images (see ﬁg. 3a).
Independent Masking + Inconsistent Cropping During data pre-processing, we can crop squareregions for input inconsistently so that images in the same temporal sequence may be spatiallyunaligned.
This strategy may help the model learn better representations as it may learn to alignimages in the sequence across the spatial and temporal dimensions.
[Figure]
4.2 Multi-spectral SatMAE
While MAE does operate on images I 2 R C ⇥ H ⇥ W , usually C = 3 for RGB images.
Satellite data,on the other hand, can often have multiple spectral bands.
For example, Sentinel-2 imagery has
C = 13 bands of 10m, 20m and 60m spatial resolution, each of different wavelengths (see A.2.2).
Below, we discuss and later experimentally compare various ways to encode spectral information.
Stack Channels
The sequence of patches S 2 R L ⇥ P 2 C is embedded to a sequence of tokens
S 0 2 R L ⇥ D , thus treating the multi-band image as is.
We denote this method SatMAE+Stack .
Group Channels There are limitations to naively stacking the spectral information, especially thata single convolutional patch embedding may be insufﬁcient to fully capture ﬁne-grained informationpresent in multiple bands of different wavelengths and spatial resolution.
We would like the model topreserve information about the different bands through the encoding and decoding stages.
To address this limitation, we propose grouping subsets of spectral bands.
Given C channels, weform G groups g 1 , g 2 , . .
, g G such that g 1 + g 2 + · · · + g G = C .
This is analogous to slicing theimage I in the channel dimension, creating images I 1 , . . .
I G , where I j 2 R g j ⇥ H ⇥ W .
We use aseparate patch embedding f p j : R P 2 G j 7!
R D for each group j , thus allowing the model to bestrepresent each possibly different group of channels as token embeddings.
Therefore, each group j isﬁrst resized from I j 2 R G j ⇥ H ⇥ W to S j 2 R L ⇥ P 2 G j , and then each patch is embedded with f p j toproduce a sequence of embedded tokens S
j 2 R L ⇥ D .
The sequences S 0 1 , . . .
, S
G are concatenatedto produce the ﬁnal set of tokens S 0 2 R GL ⇥ D .
Spectral Encoding Since the tokens in S 0 correspond to a patch location ( m, n ) in the input imageand a group of channels g j , we include an encoding for the group index k g similar to 4.1.
[Equation]
Note that this encoding simply depends on a user-devised channel grouping, and differs from eq.
(2)since additional metadata for the imagery, like its date, is not needed.
The ﬁnal encoding is aconcatenation of the positional x k,i , y k,i and the spectral encoding g k,i such that the total dimensionis D (see ﬁg. 2).
This positional encoding is added to S 0 before inputting it to the encoder.
We denotethe combined setting of grouping channels and using a group encoding as SatMAE+Group .
Masking Strategies
We consider consistent masking (denoted SatMAE+Group+CM ) and independent masking ( SatMAE+Group+IM ) as deﬁned in section 4.1.2 and as visualized in ﬁg. 3b.
Experiments
In this section, we ﬁrst introduce the datasets we considered, including a new multi-spectral remotesensing image dataset for downstream task evaluation (5.1).
We then present our results on benchmarkdatasets (5.2, 5.3, 5.4) and various remote sensing transfer-learning and downstream tasks 5.5.
Forall experiments, we compare with the current state-of-the-art methods [30, 31] and with supervisedlearning from scratch using the ViT backbone of SatMAE.
In summary, our approach demonstratesstrong performance on all the tasks we considered, yielding improvements over previous state-of-theart techniques by up to 6% on supervised learning benchmarks, and up to 14% on remote sensingtransfer-learning downstream remote sensing tasks.
5.1 Datasets for Pre-training
fMoW RGB Functional Map of the World (fMoW) [16] is a dataset of high-resolution satelliteimage time series across the world, with a task of classiﬁcation among 62 categories.fMoW
Sentinel We create a new dataset based on the fMoW RGB dataset.
We collect all 13frequency bands provided by Sentinel-2 (B1-12 and B8A) for the original fMoW locations, at someof the same times as fMoW images plus some extra times, for a total of 712,874 training images,84,939 validation images, and 84,966 test images.
More details are included in appendix A.1.
5.2 fMoW RGB (non-temporal)
[Table]
[Table]
In this section, we perform experiments onfMoW single image classiﬁcation task.
Following [30], we report both the performanceof linear probing and ﬁnetuning setting.
Table1 shows that compared to the previous stateof-the-art self-supervised method using a contrastive momentum encoding approach [30, 3],our SatMAE achieved a 6.29% improvement intop 1 classiﬁcation accuracy.
Interestingly, without SatMAE pre-training the ViT-large modelcould only reach 62.48% at convergence after50 epochs of ﬁnetuning compared to 69.05%achieved by training a ResNet-50 model fromscratch.
This is likely because the ViT [32] backbone is harder to ﬁnetune from scratch than
ResNet50 [53], which makes the pre-trainedmodel more valuable.
5.3 fMoW RGB (temporal)
Main experiments We perform image-sequence classiﬁcation on the temporal version of fMoW
RGB to evaluate our temporal SatMAE.
The temporal fMoW consists of co-located image sequenceswith a length of 3.
As seen in table 2, SatMAE surpasses the previous state-of-the-art by 4.48% andimproves the non-temporal result by 2.06% in top 1 classiﬁcation accuracy. We also outperform
UTAE [40], a SITS state-of-the-art, by 18%.
We can observe from rows 5-8 that this gain is not fromthe larger model to handle sequences of data.
Naively stacking the image sequences in the channeldimension performs even worse than the non-temporal SatMAE.
Again, the pretraining is crucial forthe ViT to achieve high performance than ResNet50.
Training details are in appendix A.3.2.
Ablation studies Table 4 provides a comprehensive ablation study on the components of ourtemporal SatMAE.
We can see that the performance mainly comes from using temporal encoding andadopting the independent masking strategy instead of consistent masking as suggested in VideoMAE[46].
Interestingly, the consistent cropping option slightly hurts the performance, indicating the modeldoes not rely on image sequences to be perfectly spatially-aligned.
In addition, applying test-timeaugmentations similar to [30] brings considerable improvement.
Further ablations on mask ratio p mand patch size P are in appendix A.4.
[Table] [Table]
[Table] [Table]
[Table]
5.4 fMoW Sentinel (Multi-spectral)
In this section
[Table]
we pretrain and ﬁnetune SatMAE on the image classiﬁcation task of the fMoWSentinel dataset.
We pre-train SatMAE+Stack 4.2 and investigate SatMAE+Group+CM 4.1.2 and
SatMAE+Group+IM 4.1.2, (see 4.2, 4.2).
The full models are then ﬁnetuned on the fMoW-Sentinelimage classiﬁcation task.
For comparison, we also ﬁnetune the ResNet-152 model [53] from scratchand from a supervised ImageNet initialization.
We pick the largest model, ResNet-152, for fairercomparison with ViTs.
We also include MoCo-v3 [54, 3], a popular SSL method.
Given thedifferences in applying RGB-image augmentations to satellite imagery, we implement two versions:(i) MoCo-v3: we apply all of the same augmentations, except random grayscale and solarize, tocreate 2 views of the 10-channel image.
(ii) MoCo-v3+Group: we split the 10 bands into two groupssuggested by [2], and apply augmentations to each to create a positive pair of two 5-channel images.
Model conﬁguration As not all of the 13 Sentinel-2 bands may be useful, in our experiments wedrop bands B1, B9 and B10, which correspond to a spatial resolution of 60m. Of the remaining10 bands, we form three groups: (i) RGB+NIR: B2, B3, B4, B8 (ii) Red Edge: B5, B6, B7, B8A(iii) SWIR: B11, B12.
We choose this grouping to ensure each group has bands of the same spatialresolution and similar wavelength (see A.2.2, A.6).
See A.3.3 for pre-training and ﬁnetuning details.
Results We present results in table 3.
Our method SatMAE+Group+IM achieves the highest accuracy, outperforming supervised training from scratch ( " 6.27%) and ImageNet-initialized backbones( " 4.84%).
ImageNet initializations may be less useful than in fMoW-RGB given the larger distributional shift to multi-spectral input data. We also note the effectiveness of grouping channels overprocessing all bands only at the patch embedding level (i.e. SatMAE+Stack).
Ablation Studies We investigate the design of SatMAE for multi-spectral data in table 5.
Forgrouping strategy, we implement alternate band groups to test the hypothesis that grouping bands
[Table]
[Table] [Table]
[Table]
based on wavelength and resolution is beneﬁcial.
X represents the band groups in 5.4.
H representssplitting the 10 bands into two halves, {(2,3,4,5,6), (7,8,8A,11,12)}.
R represents a random split intothree groups {(6,5,11,12), (8A,4,8,3), (7,2)}, reﬂecting the same group sizes as X. As seen, the choiceof band groups does inﬂuence performance, yielding a gain of about 0.6%.
Moreover, ViT-Baseperforms strongly, suggesting that SatMAE is the reason for improved performance rather than thenumber of parameters in ViT. Interestingly, independent masking performs the best, which promptsthe model to “peek” at unmasked band groups to reconstruct the same region in a masked band group.
5.5 Transfer Learning Experiments
Now, we ﬁnetune our pretrained SatMAE on downstream tasks on remote-sensing datasets, includingland cover classiﬁcation (5.5), multi-label classiﬁcation (5.5), and building segmentation (5.5).
Land Cover Classiﬁcation We perform transfer learning experiments on land cover classiﬁcationusing the NAIP and EuroSAT [55] dataset.
NAIP consists of RGB+CIR images of 66 land coverclasses obtained by the USDA’s National Agricultural Imagery Program, which are split into 100,000training and 50,000 validation images.
EuroSAT is a small dataset containing 27,000 13-band satelliteimages of 10 classes based on Sentinel-2.
We follow [31, 56] for the train/val splits on EuroSAT.
Table 6 and table 8 shows the remarkable improvement of our SatMAE over the state-of-the-arts.
Although using the ViT-Large backbone already achieved good results, initializing the model with
SAT-MAE pre-trained weights further increased the accuracy by 2%-3%.
Multi-label Classiﬁcation We also use the BigEarthNet [17] dataset for multi-label classiﬁcation,which consists of 13-band Sentinel-2 images of 19 classes in total.
There are 354,196 images fortraining and 118,065 images for validation.
Following [31], we use a 10% subset of the train set.
Table 9 shows SatMAE pre-training improves upon the model trained from scratch by over 2%, andachieves comparable results to the state-of-the-art.
GASSL and SeCo were actually trained on a largerpre-train dataset (1M Sentinel-2 images v.s. 713k) and with all 13 bands than our fMoW Sentinel.
Therefore we expect further improvement when we pre-train SatMAE with more data and for longer.
Building Segmentation
In this section, we evaluate SatMAE on the semantic segmentation downstream task of the SpaceNet v2 dataset [19].
The SpaceNet v2 datasets consists of 6940 highresolution satellite images with segmentation masks for buildings, which are divided into train andtest sets of 5000 and 1940 images, respectively.
The results in table 7 show that our method achieves a larger performance gain from supervisedlearning from scratch compared to [30].
The incompatibility of the ViT backbone with PSANet couldexplain why the baseline performance is not as strong as that of using a ResNet50 backbone.
[Figure]
We show the visualization of the reconstruction quality of two different SatMAE masking strategies inﬁg.
SatMAE+IM successfully reconstructs all the airplanes even though their number varies acrosstime.
In contrast, the SatMAE with Consistent Masking missed some airplanes in the reconstruction.
Conclusion
In this paper, we propose a new SSL framework based on the MAE architecture [1] tailored to remotesensing data (satellite imagery).
Our novel masking strategy in a joint positional, temporal/spectralspace, along with the temporal and spectral encoding, enables our model to handle temporal andmulti-spectral satellite images as input and learn useful representations.
Experiments on the datasetsfor pre-training and multiple downstream datasets demonstrate the effectiveness of our pre-trained
SatMAE model, outperforming previous state-of-the-art results by large margins.
In the future, it would be useful to design more efﬁcient transformer architectures.
While SatMAEhas a similar number of parameters for both the temporal and multi-spectral setting as a regular ViT,the increased length of token sequences can strain computational resources.
Moreover, it is also worthexploring optimal positional encodings for spectral and temporal data, as well as optimal groups ofspectral bands, either by neural-based search methods, or using prior knowledge.
Lastly, investigatingbetter architectures for object detection and semantic segmentation using ViTs will be important ingeneralising SatMAE to further downstream tasks.
Broader Impact
Accurate measurements of economic, social, and environmental phenomena are key inputs into policydecisions made around the world, but the sparsity of labelled data on many outcomes means thatsuch decisions are often not guided by timely or accurate data.
We demonstrate how a pre-trainingframework could relieve the dependence on labelled data for many downstream tasks that use satelliteimagery as input.
We hope our SatMAE method will help close the gap between SSL performanceon natural imagery and on the more challenging satellite imagery, and prompt further attention fromthe ML community on the usefulness of SSL in satellite-imagery-related tasks.
Better extraction of information from satellite imagery has profound implications for our ability tomeasure and understand a broad array of social, economic and environmental phenomena that arecritical for decision making.
Our approach further ampliﬁes the usefulness of the sparse amount oflabelled data that exist on key human outcomes, and could enable rapid and accurate extraction ofimagery features relevant for critical downstream tasks, including poverty prediction, infrastructuredevelopment, and population estimation.
Such information could aid governments in more rapid anddata-informed decision making and ultimately bring large societal beneﬁts.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] We tried our best to be precise.
(b) Did you describe the limitations of your work?
[Yes] We described the limitations inthe conclusion section.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Wediscussed in Appendix B
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will list the
url to our code repo in the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We list part of them in the experiments section and part of them in
Appendix A.
(c) Did you report error bars (e
g., with respect to the random seed after running ex-
periments multiple times)?
[No] We trained on large datasets with small variation
across different runs expected.
Limited by computation resources, we only ran each
experiment once.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We list that in Appendix A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we do that by
citing creators of datasets and authors of prior works.
(b) Did you mention the license of the assets?
[Yes] We do that in Appendix A
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We are releasing a new dataset.
The instructions and links will be released in our
codebase mentioned above.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] All data we used are released publicly.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] We concluded that no data we are using has
such concerns.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]