SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
Abstract
Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks.
Developing similar techniques for satellite imagery presents signiﬁcant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies.
In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on
Masked Autoencoder (MAE).
To leverage temporal information, we include a temporal embedding along with independently masking image patches across time.
In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneﬁcial.
Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to " 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classiﬁcation (up to " 14%) and semantic segmentation.
Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/
Introduction
In recent years self-supervised learning techniques have quickly become the norm for pre-training models on large-scale natural image datasets [1, 2, 3, 4, 5, 6, 7, 8] and have demonstrated strong performance on downstream tasks including image classiﬁcation [3, 4, 9, 10], image segmentation [3, 11], representation learning [12, 13, 14], image compression [12, 15] and image reconstruction [1].
Unlike supervised learning approaches, self-supervised learning techniques do not require human labeling, making them appealing in settings where unlabeled data are abundant but labeled data are scarce, such as remote sensing data (e g., satellite imagery).
While several large-scale satellite image datasets have been carefully curated in the past few years, including Functional
Map of the World (fMoW) [16], BigEarthNet [17], xView [18], SpaceNet [19], annotating these datasets requires specialized skills and is more expensive than traditional computer vision datasets.
Moreover, automatic analysis of satellite imagery is often needed for tasks with large societal impact such as poverty or crop yield prediction [20, 21, 22, 23, 24, 25, 26, 27, 28], where acquiring large amounts of labeled data through surveys is impossible or prohibitively expensive.
This suggests that self-supervised learning approaches for satellite imagery could be especially valuable.
However, existing self-supervised learning approaches [1, 2, 3, 4, 5, 6] are mainly designed for natural images.
As opposed to natural images such as ImageNet [29], satellite imagery is usually associated
[Table] [Table]
[Figure]
t e m p o r a l o r s p e c t r a l d a t a
[Figure]
with meaningful geographical and temporal information, and can consist of multiple spectral bands representing various sensor readings besides visible light (i.e., RGB channels typical in natural images).
Depending on the data source, satellite imagery can also vary signiﬁcantly in resolution.
While self-supervised learning methods for satellite imagery exist [30, 31], these approaches cannot learn general representations for both temporal and multi-spectral remote sensing data.
To address this issue, in this paper, we propose SatMAE , a self-supervised learning framework based on masked autoencoders (MAEs) [1] which naturally handles temporal and multi-spectral input data.
We show that introducing a positional encoding for the temporal/spectral dimension and independently masking patches across the temporal/spectral dimension beneﬁts pre-training, allowing the model to learn representations of the data that are more conducive to ﬁne-tuning.
Speciﬁcally, our contributions are as follows
We propose a novel method to leverage temporal or multi-spectral information in satellite imagery to improve self-supervised pre-training with masked autoencoders (see 4).
We introduce fMoW-Sentinel, a new Sentinel-2 dataset cross-referenced with fMoW, as a benchmark for training models on multi-spectral satellite imagery (see 5.1).
We demonstrate the effectiveness of pre-training transformers [32] on satellite imagery, achieving signiﬁcant improvement over previous state-of-the-art methods on benchmark datasets as well as downstream remote sensing tasks (see 5)
Related Work
ML for SITS Deep learning has been used for many Satellite Image Time Series (SITS) supervisedlearning tasks such as crop-type mapping [28, 27, 33, 34], yield prediction [35, 36], precipitation forcasting [37], land-cover classiﬁcation [38, 39, 40, 26].
These works establish the usefulness of tailoring architectures such as LSTMs, self-attention, and transformers to temporal data.
However, outside of their speciﬁc task, they are often not directly applicable to other remote-sensing datasets.
SSL for Satellite Imagery Self-supervised learning [2, 3, 4, 5, 6] has emerged as a promising approach in remote sensing domains.
For instance, [30] and [31] propose incorporating spatially aligned images over time for contrastive self-supervised learning.
Despite promising results, these two contrastive learning approaches rely heavily on the quality of positive pairs, which is often hard to control.
[41] combines different sensor channels to generate co-located images that serve as positive pairs.
[42, 43, 44] apply off-the-shelf contrastive learning algorithms to satellite images.
[44] utilizes image inpainting and transformation prediction as additional pretext tasks.
[45] leverages geographical knowledge to aid SSL, which, however, can be difﬁcult to obtain as annotations.
Masked Autoencoder MAE [1] is a recent powerful self-supervised learning method.
Instead of constructing a contrastive objective, it proposes the pretext task of reconstructing masked patches of the input, and largely avoids the need for designing speciﬁc data augmentation.
Inspired by
MAE’s state-of-the-art performance on a wide collection of vision benchmarks [1], many follow-up works extend MAE to different data modalities.
VideoMAE [46] proposes video tube masking and reconstruction as a pretext task for video analysis.
GMAE [47] adapts MAE to the domain of graphs.
MultiMAE [48] takes optional inputs of different modalities and accordingly includes other training objectives to facilitate multi-modality learning.
However, these works fail to optimally handle temporal and multi-spectral input.
VideoMAE requires equally-spaced image frames in the temporal dimension, which is not the case for satellite data given the temporal irregularity and discontinuity in sampling images of a location.
In this work, we incorporate temporal and spectral information into a masked autoencoder architecture, and propose a novel self-supervised framework for satellite data.
Background
Masked Autoencoder The MAE is an autoencoder with asymmetrical encoding and decoding stages [1].
It operates on images I 2 R C ⇥ H ⇥ W , where H, W are the height and width of the image, respectively, and C is the number of channels.
The input image I is resized to a sequence of non-overlapping patches, S 2 R L ⇥ P 2 C , where P is the height and width of the patch, and
L =
( H/P ) · ( W/P ) is the number of patches.
Each patch is passed through a patch embedding f p : R P 2 C 7!
R D to create a sequence S 0 2 R L ⇥ D of embedded patch “tokens”.
A fraction p m of the L tokens are masked and only the remaining (1  p m ) L “visible" patch tokens are fed to the encoder, a Vision Transformer (ViT)
[32] with positional embeddings to capture the spatial location of the patch in the image.
The decoder is a series of transformer blocks that operates on all L tokens (with positional embeddings added), where the p m L encoded visible patches are placed in their original sequence position among (1  p m )
L masked patches represented by a learnable mask token.
The decoder outputs a reconstructed image ˆ I 2 R C ⇥ H ⇥ W , which is compared to the original image using the mean-squared error (MSE) loss, computed per-pixel only on the masked patches [1].
Positional encoding Positional encoding allows transformers to make their learned representations position-aware.
In MAE [1] and in many transformers [49, 50], the positional encoding is:
[Equation]
Here, k is the position, i is the index of feature dimension in the encoding, d is the number of possible positions, and ⌦ is a large constant (normally set to 10000).
In MAE, position is deﬁned as the index of the patch along the x or y axes.
Therefore, k ranges from 0 to H/P (or W/P ).
The ﬁnal encoding is generated by concatenating the encodings of the x and y coordinates.
Method
In this section, we describe SatMAE with temporal (4.1) and multi-spectral (4.2) satellite images.
4.1 Temporal SatMAE
We now consider input tensors I T 2 R T ⇥ C ⇥
H ⇥ W , where T denotes the number of images in a temporal sequence.
In video data, T frames are usually equally spaced.
However, temporal satellite imagery rarely has images at regular intervals.
More commonly, several snapshots, or versions, of a given location are taken at irregular times.
The length and sample frequency of these sequences of satellite images vary drastically over years and across different regions.
Naïvely, one could reshape I T to I
T 2 R TC ⇥
H ⇥ W , effectively concatenating the temporal sequence of images along the spectral (i.e. channel) dimension, and then apply the MAE machinery verbatim.
This method poses a few difﬁculties: (i) the model may be unable to generalise to a temporal ordering different to the one used in pre-training, since it can only understand order through the position of images in the stacked-timeseries (ii) the model cannot reason about the length of time separating two consecutive images in a time sequence, which may be variable when images of a location are sampled at irregular intervals (iii) the model loses access to temporal ﬁne-grained information in deeper layers, as its only direct exposure to encode temporal information is through the initial patch embedding f p (iv) the model is not temporally-shift invariant (i.e. the model would need to separately learn to detect the same event in two different segments of a temporal sequence).
To address these challenges and to avoid losing temporal information, we resize the temporal sequence
I T to S T 2 R L T ⇥ P T P 2 C , where L T =
L · ( T/P T ) =
( H/P ) · ( W/P ) · ( T/P T ) , P T is the “patch size” in the temporal dimension, and L and P are deﬁned in 3.
Prior works using transformers for video data suggest using P T = 2 , where each “patch” is a cube of shape 2 ⇥ 16 ⇥ 16 [46, 51, 52].
Since our data has much shorter temporal sequence lengths [16], we let P T = 1 such that L T =
L · T .
In order to operate on inputs of any temporal order, we re-use the same patch embedding f p : R P 2 C 7!
R D for each image in the time series, giving us an embedded sequence of
[Figure]
S
[Figure]
4.1.1 Temporal Encoding
For each embedded token in the L T length sequence, we need to ensure the model retains information about its spatial and temporal position.
As shown in many prior works [30, 31], the timestamp of a satellite image is useful for many pre-training or downstream vision tasks.
We propose a temporal encoding scheme compatible with the masked autoencoder architecture by treating the temporal dimension similarly to the positional dimensions (see 3).
The timestamp of a satellite image is represented as “yearmonth-day-hour-minute-second”.
Instead of passing the entire numerized timestamp into a feature encoder, we propose only keeping the useful parts.
Intuitively, the day, minute, and second should be unrelated to the visual appearance of a region.
Thus, including these components in the temporal encoding may not be beneﬁcial, and can even be detrimental.
In contrast, a landscape may evolve over years due to weather, geology, and human activity.
The month reﬂects season and climate, and the hour reﬂects daylight and temperature.
Then, the temporal encoding is formulated as:
[Equation]
And the ﬁnal encoding is generated by concatenating the temporal encoding to the positional encoding deﬁned in 3 such that the total length of the encoding is D .
4.1.2 Masking Strategies
With an additional temporal dimension, masking a subset of the L T tokens needs to be treated with care.
As seen in ﬁgure 3, there are different ways to mask a temporal series of satellite images.
Consistent Masking Each image is “patchiﬁed” separately, but the masked regions are consistent across all images (ﬁg. 3a).
This approach is also used in VideoMAE [46], with video input.
Independent Masking
Each image is “patchiﬁed” separately, and masked regions may not be the same across every image.
Instead, a fraction p m of the full sequence of all patch tokens are masked.
Another variant is to independently mask the regions of each image, but keep the ratio p m of masked regions ﬁxed per image.
Both variants are equivalent in expectation.
Effectively, the model may look at unmasked values of a region that is masked in one image but not in others.
This setting may lead to an easier task for video data since the model can “cheat” and exploit temporal redundancy in videos with high framerates [46].
However, we argue that this form of “cheating” is less feasible in temporal
[Figure]
satellite imagery, given the strong impact of seasonal variation and changing human activity over periods of time and the much larger time deltas between temporally consecutive images (see ﬁg. 3a).
Independent Masking + Inconsistent Cropping During data pre-processing, we can crop square regions for input inconsistently so that images in the same temporal sequence may be spatiallyunaligned.
This strategy may help the model learn better representations as it may learn to align images in the sequence across the spatial and temporal dimensions.
4.2 Multi-spectral SatMAE
While MAE does operate on images I 2 R C ⇥ H ⇥ W , usually C = 3 for RGB images.
Satellite data, on the other hand, can often have multiple spectral bands.
For example, Sentinel-2 imagery has
C = 13 bands of 10m, 20m and 60m spatial resolution, each of different wavelengths (see A.2.2).
Below, we discuss and later experimentally compare various ways to encode spectral information.
Stack Channels
The sequence of patches S 2 R L ⇥ P 2 C is embedded to a sequence of tokens
S 0 2 R L ⇥ D , thus treating the multi-band image as is.
We denote this method SatMAE+Stack .
Group Channels There are limitations to naively stacking the spectral information, especially that a single convolutional patch embedding may be insufﬁcient to fully capture ﬁne-grained information present in multiple bands of different wavelengths and spatial resolution.
We would like the model to preserve information about the different bands through the encoding and decoding stages.
To address this limitation, we propose grouping subsets of spectral bands.
Given C channels, we form G groups g 1 , g 2 , . . .
, g G such that g 1 + g 2 + · · · + g G = C .
This is analogous to slicing the image I in the channel dimension, creating images I 1 , . . .
I G , where I j 2 R g j ⇥ H ⇥ W .
We use a separate patch embedding f p j : R P 2 G j 7!
R D for each group j , thus allowing the model to best represent each possibly different group of channels as token embeddings.
Therefore, each group j is ﬁrst resized from I
j 2 R G j ⇥ H ⇥ W to S j 2 R L ⇥ P 2 G j , and then each patch is embedded with f p j to produce a sequence of embedded tokens S
j 2 R L ⇥ D .
The sequences S 0 1 , . . .
, S 0 G are concatenated to produce the ﬁnal set of tokens S 0 2 R GL ⇥ D .
Spectral Encoding Since the tokens in S 0 correspond to a patch location ( m, n ) in the input image and a group of channels g j , we include an encoding for the group index k g similar to 4.1.
[Equation]
Note that this encoding simply depends on a user-devised channel grouping, and differs from eq.
(2) since additional metadata for the imagery, like its date, is not needed.
The ﬁnal encoding is a concatenation of the positional x k,i , y k,i and the spectral encoding g k,i such that the total dimension is D (see ﬁg. 2).
This positional encoding is added to S 0 before inputting it to the encoder.
We denote the combined setting of grouping channels and using a group encoding as SatMAE+Group .
Masking Strategies
We consider consistent masking (denoted SatMAE+Group+CM ) and independent masking ( SatMAE+Group+IM ) as deﬁned in section 4.1.2 and as visualized in ﬁg. 3b.
Experiments
In this section, we ﬁrst introduce the datasets we considered, including a new multi-spectral remote sensing image dataset for downstream task evaluation (5.1).
We then present our results on benchmark datasets (5.2, 5.3, 5.4) and various remote sensing transfer-learning and downstream tasks 5.5.
For all experiments, we compare with the current state-of-the-art methods [30, 31] and with supervised learning from scratch using the ViT backbone of SatMAE.
In summary, our approach demonstrates strong performance on all the tasks we considered, yielding improvements over previous state-of-theart techniques by up to 6% on supervised learning benchmarks, and up to 14% on remote sensing transfer-learning downstream remote sensing tasks.
5.1 Datasets for Pre-training
fMoW RGB Functional Map of the World (fMoW) [16] is a dataset of high-resolution satellite image time series across the world, with a task of classiﬁcation among 62 categories.
fMoW Sentinel We create a new dataset based on the fMoW RGB dataset.
We collect allfrequency bands provided by Sentinel-2 (B1-12 and B8A) for the original fMoW locations, at some of the same times as fMoW images plus some extra times, for a total of 712,874 training images, 84,939 validation images, and 84,966 test images.
More details are included in appendix A.1.
5.2 fMoW RGB (non-temporal)
[Table] [Table]
[Table] [Table]
In this section, we perform experiments on fMoW single image classiﬁcation task.
Following [30]
[Table]
we report both the performance of linear probing and ﬁnetuning setting.
Table 1 shows that compared to the previous stateof-the-art self-supervised method using a contrastive momentum encoding approach [30, 3], our SatMAE achieved a 6.29% improvement in top 1 classiﬁcation accuracy.
Interestingly, without SatMAE pre-training the ViT-large model could only reach 62.48% at convergence after 50 epochs of ﬁnetuning compared to 69.05% achieved by training a ResNet-50 model from scratch.
This is likely because the ViT [32] backbone is harder to ﬁnetune from scratch than
ResNet50 [53], which makes the pre-trained model more valuable.
5.3 fMoW RGB (temporal)
Main experiments We perform image-sequence classiﬁcation on the temporal version of fMoW
RGB to evaluate our temporal SatMAE.
The temporal fMoW consists of co-located image sequences with a length of 3.
As seen in table 2, SatMAE surpasses the previous state-of-the-art by 4.48% and improves the non-temporal result by 2.06% in top 1 classiﬁcation accuracy.
We also outperform
UTAE [40], a SITS state-of-the-art, by 18%.
We can observe from rows 5-8 that this gain is not from the larger model to handle sequences of data.
Naively stacking the image sequences in the channel
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
dimension performs even worse than the non-temporal SatMAE.
Again, the pretraining is crucial for the ViT to perform better than ResNet50.
Training details are in appendix A.3.
Ablation studies Table 4 provides a comprehensive ablation study on the components of our temporal SatMAE.
We can see that the performance mainly comes from using temporal encoding and adopting the independent masking strategy instead of consistent masking as suggested in VideoMAE [46].
Interestingly, the consistent cropping option slightly hurts the performance, indicating the model does not rely on image sequences to be perfectly spatially-aligned.
In addition, applying test-time augmentations similar to [30] brings considerable improvement.
Further ablations on mask ratio p m and patch size P are in appendix A.4.
5.4 fMoW Sentinel (Multi-spectral)
In this section, we pretrain and ﬁnetune SatMAE on the image classiﬁcation task of the fMoWSentinel dataset.
We pre-train SatMAE+Stack 4.2 and investigate SatMAE+Group+CM 4.1.2 and
SatMAE+Group+IM 4.1.2, (see 4.2, 4.2).
The full models are then ﬁnetuned on the fMoW-Sentinel image classiﬁcation task.
For comparison, we also ﬁnetune the ResNet-152 model [53] from scratch and from a supervised ImageNet initialization.
We pick the largest model, ResNet-152, for fairer comparison with ViTs.
We also include MoCo-v3 [54, 3], a popular SSL method.
Given the differences in applying RGB-image augmentations to satellite imagery, we implement two versions: (i) MoCo-v3: we apply all of the same augmentations, except random grayscale and solarize, to create 2 views of the 10-channel image.
(ii) MoCo-v3+Group: we split the 10 bands into two groups suggested by [2], and apply augmentations to each to create a positive pair of two 5-channel images.
[Table]
[Table] [Table]
[Table]
Model conﬁguration As not all of the 13 Sentinel-2 bands may be useful, in our experiments we drop bands B1, B9 and B10, which correspond to a spatial resolution of 60m. Of the remaining 10 bands, we form three groups: (i) RGB+NIR: B2, B3, B4, B8 (ii) Red Edge: B5, B6, B7, B8A (iii) SWIR: B11, B12.
We choose this grouping to ensure each group has bands of the same spatial resolution and similar wavelength (see A.2.2, A.6).
Only the last row of table 3 includes additional data augmentations used during ﬁnetuning as in [1].
See A.3.3 for pre-training and ﬁnetuning details.
Results We present results in table 3.
Our method SatMAE+Group+IM achieves the highest accuracy, outperforming supervised training from scratch ( " 6.27%) and ImageNet-initialized backbones ( " 4.84%).
ImageNet initializations may be less useful than in fMoW-RGB given the larger distributional shift to multi-spectral input data.
We also note the effectiveness of grouping channels over processing all bands only at the patch embedding level (i.e. SatMAE+Stack).
Ablation Studies We investigate the design of SatMAE for multi-spectral data in table 5.
For grouping strategy, we implement alternate band groups to test the hypothesis that grouping bands based on wavelength and resolution is beneﬁcial.
X represents the band groups in 5.4.
H represents splitting the 10 bands into two halves, {(2,3,4,5,6), (7,8,8A,11,12)}.
R represents a random split into three groups {(6,5,11,12), (8A,4,8,3), (7,2)}, reﬂecting the same group sizes as X. As seen, the choice of band groups does inﬂuence performance, yielding a gain of about 0.6%.
Moreover, ViT-Base performs strongly, suggesting that SatMAE is the reason for improved performance rather than the number of parameters in ViT. Interestingly, independent masking performs the best, which prompts the model to “peek” at unmasked band groups to reconstruct the same region in a masked band group.
5.5 Transfer Learning Experiments
Now, we ﬁnetune our pretrained SatMAE on downstream tasks on remote-sensing datasets, including land cover classiﬁcation (5.5), multi-label classiﬁcation (5.5), and building segmentation (5.5).
Land Cover Classiﬁcation We perform transfer learning experiments on land cover classiﬁcation using the NAIP and EuroSAT [55] dataset.
NAIP consists of RGB+CIR images of 66 land cover classes obtained by the USDA’s National Agricultural Imagery Program, which are split into 100, training and 50,000 validation images.
EuroSAT is a small dataset containing 27,000 13-band satellite images of 10 classes based on Sentinel-2.
We follow [31, 56] for the train/val splits on EuroSAT.
Table 6 and table 8 shows the remarkable improvement of our SatMAE over the state-of-the-arts.
Although using the ViT-Large backbone already achieved good results, initializing the model with
SAT-MAE pre-trained weights further increased the accuracy by 2%-3%.
Multi-label Classiﬁcation We also use the BigEarthNet [17] dataset for multi-label classiﬁcation, which consists of 13-band Sentinel-2 images of 19 classes in total.
There are 354,196 images for training and 118,065 images for validation.
Following [31], we use a 10% subset of the train set.
Table 9 shows SatMAE pre-training improves upon the model trained from scratch by over 2%, and achieves comparable results to the state-of-the-art.
GASSL and SeCo were actually trained on a larger pre-train dataset (1M Sentinel-2 images v.s. 713k) and with all 13 bands than our fMoW Sentinel.
Therefore we expect further improvement when we pre-train SatMAE with more data and for longer.
Building Segmentation
In this section, we evaluate SatMAE on the semantic segmentation downstream task of the SpaceNet v1 dataset [19].
The SpaceNet v1 dataset consists of 6940 high resolution satellite images with segmentation masks for buildings, which are divided into train and test sets of 5000 and 1940 images, respectively.
The results in table 7 show that our method achieves a larger performance gain from supervised learning from scratch compared to [30].
The incompatibility of the ViT backbone with PSANet could explain why the baseline performance is not as strong as that of using a ResNet50 backbone.
[Figure]
We show the visualization of the reconstruction quality of two different SatMAE masking strategies in ﬁg. 4.
SatMAE+IM successfully reconstructs all the airplanes even though their number varies across time.
In contrast, the SatMAE with Consistent Masking missed some airplanes in the reconstruction.
Conclusion
In this paper, we propose a new SSL framework based on the MAE architecture [1] tailored to remotesensing data (satellite imagery).
Our novel masking strategy in a joint positional, temporal/spectral space, along with the temporal and spectral encoding, enables our model to handle temporal and multi-spectral satellite images as input and learn useful representations
Experiments on the datasets for pre training and multiple downstream datasets demonstrate the effectiveness of our pre- trained
SatMAE model, outperforming previous state of-the-art results by large margins.
In the future, it would be useful to design more efﬁcient transformer architectures.
While SatMAE has a similar number of parameters for both the temporal and multi-spectral setting as a regular ViT, the increased length of token sequences can strain computational resources.
Moreover, it is also worth exploring optimal positional encodings for spectral and temporal data, as well as optimal groups of spectral bands, either by neural-based search methods, or using prior knowledge.
Lastly, investigating better architectures for object detection and semantic segmentation using ViTs will be important in generalising SatMAE to further downstream tasks.
Broader Impact
Accurate measurements of economic, social, and environmental phenomena are key inputs into policy decisions made around the world, but the sparsity of labelled data on many outcomes means that such decisions are often not guided by timely or accurate data.
We demonstrate how a pre-training framework could relieve the dependence on labelled data for many downstream tasks that use satellite imagery as input.
We hope our SatMAE method will help close the gap between SSL performance on natural imagery and on the more challenging satellite imagery, and prompt further attention from the ML community on the usefulness of SSL in satellite-imagery-related tasks.
Better extraction of information from satellite imagery has profound implications for our ability to measure and understand a broad array of social, economic and environmental phenomena that are critical for decision making.
Our approach further ampliﬁes the usefulness of the sparse amount of labelled data that exist on key human outcomes, and could enable rapid and accurate extraction of imagery features relevant for critical downstream tasks, including poverty prediction, infrastructure development, and population estimation.
Such information could aid governments in more rapid and data-informed decision making and ultimately bring large societal beneﬁts.
Acknowledgements
This research is based upon work supported in part by the Ofﬁce of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2021-2011000004,
HAI, NSF(#1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-0125) and Sloan Fellowship.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes not-withstanding any copyright annotation therein.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We tried our best to be precise.
(b) Did you describe the limitations of your work?
[Yes] We described the limitations in the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discussed in Appendix B (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We list the url to our project website in the abstract.
The website will contain links to the code and data on Github.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We list part of them in the experiments section and part of them in Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We trained on large datasets with small variation across different runs expected.
Limited by computation resources, we only ran each experiment once.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We list that in Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we do that by citing creators of datasets and authors of prior works.
(b) Did you mention the license of the assets?
[Yes] We do that in Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We are releasing a new dataset.
The instructions and links will be released in our codebase mentioned above.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] All data we used are released publicly.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] We concluded that no data we are using has such concerns.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]